[…] is the second post in a series of blog posts based on my thesis. Read the introductory post here which describes the developmental […]

[…] Developmental appropriateness as cultural competence in evaluation […]

Background

On May 25, 2020, George Floyd was killed by police officers.
This news is not new. George’s cries of “I can’t breathe” echo the same cries
by Eric Garner, who was also killed by police officers on July 17, 2014. Black,
indigenous, and people of color (BIPOC) are all disproportionately more likely to be stopped,
arrested, incarcerated, and killed by police than White people: indigenous
people are 3.5 times more likely and Black people are 3 times more likely to
be killed by police than White people.

The problem is not just with police or prisons. Segregation still exists, with both racially homogeneous schools and neighborhoods. Black and Latinx families are less likely to live in owner-occupied housing compared with White families. Voter suppression disproportionately affects BIPOC. The employment rate is around 15% lower for Black people than for White people. The end of affirmative action policies across states and colleges have led to significant declines in the number of BIPOC at large state schools. Black and Brown students are much less likely to graduate college with a degree compared to White and Asian students. People of color have poorer health outcomes (e.g., diabetes, heart disease, worse life expectancy) and are more likely to die from Covid-19. The list goes on.

I bring up this information to show the racial disparities
present in a variety of areas: health, education, voting, policing and incarceration,
housing, and more. These areas are many of those that we evaluators study.
And those programs and policies disproportionately
serve BIPOC. When a white person evaluates a program serving BIPOC, then
there is even greater need for a culturally responsive evaluation approach.

Culturally Responsive Evaluation

In 2011 the American Evaluation Association issued a statement on cultural competence in
evaluation. In it, they describe that culture affects all parts of an
evaluation, and “evaluations cannot be culture free.” It is part of our ethical
duty as evaluators to attend to culture in our evaluations and to make valid
inferences. However, to do so requires a “shared understanding within and
across cultural contexts.”

Although an evaluator can develop such a shared
understanding through deliberate practice and training, a number of
studies in cross-cultural evaluation suggest that culturally competent
evaluation is best done by evaluators who share the same cultural identity as
the evaluands. Therefore, the field of evaluation should focus on training
and promoting BIPOC evaluators so that our field can promote culturally
responsive evaluation.

Indeed, culturally responsive evaluation should be the default
evaluation approach. If culture is everywhere, and no evaluation can be culture-free,
then culturally responsive evaluation is good evaluation. If we truly
care about the validity of our evaluation findings, then we need to employ
culturally responsive evaluation practices in all our evaluations.

White Culture in Evaluation

Yet this can be difficult for White evaluators like myself.
We naïvely believe that we have no culture, but we come to that conclusion
because the White culture is the dominant culture in society. White culture has
become the one in power, the one that becomes the “default” and the “norm.”

What are the characteristics
of a White dominant culture? Perfectionism, a sense of urgency,
defensiveness, quantity over quality, worship of the written word, paternalism,
either/or thinking, power hoarding, fear of open conflict, individualism,
objectivity, and a right to comfort for those in power. One need only peruse
EvalTalk to see all these characteristics come full bloom once the White
dominant culture is threatened (as evident in the early June 2020 conversations).

When White culture is the dominant culture, then the
evaluations designed by White evaluators become the default evaluation. If you
want to see the White dominant frame in evaluation, look at our predominantly
White evaluation theory tree. Look out who we call the “fathers” of evaluation.
Look at who is being cited (and,
therefore, read). Look at how we call non-White evaluation culturally
responsive evaluation (instead of just evaluation).

When AEA put together the Race and Class Dialogues in 2017,
there was incredibly low attendance across the sessions. The 2018
Member Survey revealed that of the 1,484 respondents who answered the
question 582 (39%) were not aware of them and 629 (42%) did not participate. Of
the 273 (%) that did respond, 15.8% found them “not at all useful,” 29% said
they were “somewhat useful,” and 55% said they were “useful.” If evaluators
were not oblivious, then they decided not to participate. And for those that
did participate, nearly half decided they were not very useful, despite our
statement on cultural competence. However, further reading of the 2018 Member
Survey shows that 18% of members were not aware of the statement on cultural
competence, 24% had not read it, but at least most who did take the time to
read it found it useful (68%). White evaluators need to become familiar with
these resources and incorporate them into their regular evaluation practices.

Stepping Up

We live in a society[1] in
which the dominant frame is that of the White, male, Christian, heterosexual,
well-educated, affluent, and able-bodied person. Each of us may identify with
none, one, some, or all of these dominant frames, and each of these dominant frames have
a form of oppression associated with it: racism, sexism, religious oppression,
heterosexism, classism, ageism, and ableism.

We should first recognize our identities for what they are.
For example, I have the privileges and systematically receive the benefits of
being a White, heterosexual, well-educated, middle-class, and able-bodied
person. I may be systematically disadvantaged by my gender, and perhaps my age,
but my privileges are numerous. And because these privileges are part of the
dominant culture, they are often taken for granted and considered “normal.”

It is the responsibility of the privileged to use that power
and privilege. Too often, we have placed the burden of change on those
experiencing the oppression. There are many things we can do to improve our
evaluation practices as White evaluators:

1. Learn about Your White Privilege

Take time to learn about the privileges you hold. This blog
post focuses on white privilege, and so here are some resources to begin
understanding your white privilege. The first step we can take as White
evaluators is to understand our White dominant culture.

Here are a list of books I have read or are highly
recommended by my evaluator colleagues:

1. White
   Fragility: Why It’s So Hard for White People to Talk About Racism by Robin
   DiAngelo
2. How to Be An
   Antiracist by Dr. Ibram X. Kendi
3. So
   You Want to Talk About Race by Ijeoma Oluo
4. Why
   Are All the Black Kids Sitting Together in the Cafeteria? And Other
   Conversations About Race by Beverly Daniel Tatum
5. Me
   and White Supremacy: Combat Racism, Change the World, and Become a Good
   Ancestor by Layla F. Saad
6. Witnessing
   Whiteness: The Need to Talk About Race and How to Do It by Shelly Tochluk
7. My
   Grandmother’s Hands: Racialized Trauma and the Pathway to Mending Our Hearts
   and Bodies by Resmaa Menakem

If you’re interested in podcasts, articles, films and TV
series, or organizations to follow, I highly recommend this
resource.

I also recommend building racial equity habits. Eddie Moore
Jr. and Debby Irving created a 21-day racial equity habit
building challenge. They suggest for 21 days that you do at least one
action to further your understanding of power, privilege, supremacy,
oppression, and equity. They provide suggestions in the categories of things to
read, listen, watch, notice, connect, engage, act, reflect, and stay inspired. Although
they named it a 21-day challenge, it does not have to be 21 days nor do the
days have to be continuous. Just like we can never become culturally competent
(rather we can only do culturally competent actions), our work here is never
done.

2. Learn about and incorporate Culturally Responsive Evaluation Practices

The AEA statement on cultural competence points to a need to
have a shared understanding. Although evaluators who share the culture of the
program and its participants are best positioned to do culturally responsive
evaluation, White evaluators can still deliberately practice and receive
training to better do culturally responsive evaluation in contexts they do not
share the culture.

There is a great deal of humility needed to do this type of
work. A humility that the program knows what is best for them. A humility to
set aside the White dominant culture and all the assumptions and norms you have
internalized from birth. It is hard work, and you will mess up. Think of
yourself as a beginner in this work and be open to learning.

3.  Critically reflect on your
evaluation practices

As you are incorporating culturally responsive evaluation
practices, you should be attending to the defaults, assumptions, and norms you
have carried into your evaluation practice. You will have many thoughts and
feelings going through this process. Create a space with other White
evaluators to discuss these thoughts and feelings and help each other through
this process. Do not rely on your BIPOC colleagues to help you through
this. They have told us for years, and they tell us now they are tired. Listen
to them and do your own work.

Some ideas of questions and comments you may want to ponder
regarding your evaluation practices:

1. Think critically about for whom in your
   evaluation work. For whom does the program work? For whom does the program unintentionally
   (or intentionally!) harm?
2. When you think of evaluation of a program, what
   comes to mind? When you think of evaluation of the sameprogram
   serving BIPOC, what comes to mind? Are they the same, or different? What do
   you think that says about you and your evaluation practice?
3. If you work with other evaluators, who are you hiring?
   Do they primarily look like you? Does your evaluation team share the lived
   experiences of the people in the programs you are working with?
4. Who do you include in the evaluation process? Madison
   concluded in her review of 20 years of New Directions for Evaluation volumes
   that “underrepresented groups continue to be presented as subjects of
   evaluation rather than as invested stakeholders.” The stakeholders most often
   to be included in the evaluation consist primarily of the program leaders who,
   like evaluators, are less likely to
   be BIPOC.
5. What choices do you routinely make in your
   evaluation practices? Some major choices include the design and methods you
   use. For each of those choices, examine how race, class, gender, sexual
   orientation, and more affect the decisions you make. Refrain from thinking that
   they do not make a difference. For example, the creators
   of modern statistics did so to promote a eugenics agenda and data is not value-neutral,
   equitable, or unbiased.

4. Critically Reflect on the Field of Evaluation

Recognize that racism is everywhere, including at the AEA conference,
on EvalTalk, and more. BIPOC evaluators have called out evaluation spaces for
being racist and unsafe spaces for them for years. Start listening to them,
believing them, and supporting them.

The AEA conference in 2019 made one step forward when it
offered “A
Respite & Healing Space” for BIPOC attendees to rest, heal, and connect
with other BIPOC attendees. White allies and accomplices were invited to
contribute financially as their way to support the space. Listen and act when
other opportunities are presented for you to show your allyship for our BIPOC
colleagues.

If you are a teacher in evaluation, think critically
about who you accept as students and what you teach. Recruit and accept BIPOC
students into your program. Encourage your institution to hire and create
safe work environments for BIPOC. Check your syllabus: who do you assign as
readings? What topics are covered? How are race, class, gender, and other
topics infused into your curriculum? A week on culturally responsive evaluation
is a starting point, but it is not enough.

If you are a researcher of evaluation, think critically
about who you cite. Our BIPOC colleagues are not cited at the same rates as
White evaluators. Check your reference list that you have a balance of
representation by race, gender, nationality, and more.  

Conclusion

Listening, stepping back, and critically reflecting is hard work. But what is harder and just as necessary is for us to speak up, stand up, and step up in alliance of our BIPOC colleagues. Advocate for change in your institution, in our association, and in our field in general. Advocate for change with the organizations you partner with, citing our guiding principles and statement of cultural competence when you write an evaluation proposal. Speak up when you see prejudiced behaviors or racist systems. It is time. Step up.

Do you have comments, questions, feedback, or suggestions? Add a comment below or use the contact form to email me on my website.

[1] I focus on the United States in this blog post because that is the nation I know best, but this is likely true for other nations as well. I mention because Khalil Bitar rightfully pointed out that these conversations are largely USA-centered, mostly reactive, and there is limited participation by the often unheard voices.

Packing up the spring 2020 semester and transitioning to
remote teaching was difficult, not just for me but also for my students. As I
told them, “None of us signed up for online teaching. But we’ll make it through
this.” And we did! At least most of us did…

Part of the challenge was doing this in my first year at
UW-Stout. I was prepping two courses (undergraduate stats and graduate
evaluation), and practically prepping my two sections of intro psychology after
a major overhaul from the fall. I was trying to keep up my scholarship and
service commitments. I was already a little burned out from the semester and
had to skip my spring break to focus on transitioning to remote teaching. I was
trying to figure out how to work from home again after finally getting used to
working in my department every day.

On reflection, I think there were some things I did really
well with the transition that I want to keep in mind for future semesters.
There were also some things I struggled with and want to improve for next time.
Here are the five things I am taking away from my experiences with remote teaching
due to Covid-19:

1. 
Have more lenient late policies

I already had a fairly lenient late policy: the points
possible decrease every day an assignment is late, then it’s maximum 50% credit
but you can turn it in at any time during the semester.

After the transition, I made it even more lenient: turn in
anything for full credit by the last day of the semester. In other words, there
was no late policy. Deadlines were all suggestions.

Students were grateful for both structure of assignment
deadlines and for leniency when they couldn’t meet them. Some students needed
the regular deadlines, whereas others needed to focus on other courses before
they could think about mine. And you know what? It worked out just fine.

I was always a little worried about abuses to the policy. Would
students get the answers from another student to submit their homework? Would I
get a huge influx of grading right at the end of the semester? Neither seemed
to happen.

This policy doesn’t work for all situations, particularly
when a large project is broken up into multiple sub-projects. But I plan on
being much more lenient in the future.

Relatedly, I’m going to think more critically about
high-stakes assignments. All my classes went to low-stakes assignments
throughout the semester, and it was much more enjoyable for both students and
myself.

2. 
Incorporate more videos and flip the classroom

For my undergraduate stats class, I recorded all my attendance-optional
lectures and put them into our LMS. This didn’t require any additional effort
or time beyond adding the link to the LMS after the recording was done.

Yet it saved me so much time answering student questions. Students
learned pretty quickly that most of their questions could be answered by
referring back to the lecture, so I wasn’t fielding as many repetitive
questions about the basics. Instead, I could focus on the more advanced
questions students were asking about the content we were learning. It was so
much more rewarding!

I will definitely be incorporating more videos into my
classes in upcoming semesters, regardless of whether we’re online or in person.
I want to design my courses to be disaster-ready: flipped so that much of the
learning is on their own and class time is spent applying the content. That
way, if something does happen, they’re already set to finish out the semester
the way they started the semester.

p.s. If you’re looking to improve your videos, I finished
reading Karen Costa’s “99
Tips for Creating Simple and Sustainable Education Videos” and I highly
recommend it.

3. 
Build a better classroom community

I was particularly proud of the community that I build in my
intro psych classes using team-based learning. But Covid-19 hit just when teams
were starting to norm and perform. To accommodate students’, I went completely
asynchronous and made the class as easy as possible for them to complete (and
still I had a number who sadly were not able to complete).

But the one thing I struggled with was keeping up the
classroom community we built when we were in-person. This was a struggle for
all my classes, even my graduate class that kept up synchronous meetings. I
just felt like I wasn’t connecting with a lot of my students anymore. I couldn’t
check in with them before, during, or after class like I was doing prior. And
email check-ins just weren’t the same…

I’m still not entirely sure how to go about this. I have
some ideas though: more videos to humanize myself and connect with students,
especially if we’re online; having an assignment for points that has students
come visit me during student hours, whether that be in my office or online;
continue to use MS Teams for each of my classes and grading on participation; and
continue to hold online student hours, even if I have in-person student hours. I’d
love other suggestions you have for building a community in an online
environment.

4. 
Embrace universal design for learning

The transition to remote teaching made me better embrace UDL in my classes: providing multiple
means of engagement, representation, and action and expression. I was already
doing it a little bit, but I realized how important it was for my students that
they have alternative ways to access and participate in the learning opportunities.
  

For example, I had two small papers in my intro psych class.
At the beginning of the semester, I decided to open it up so that students
could either write an essay or they could record a video. After the transition,
I had a student ask to just call me and describe what they did, which of course
I allowed.

I still have a lot to learn about UDL—and the CAST website provides a ton of great guidelines and resources—but one of my core principles in teaching is that my teaching is accessible for all students.

What about you? What lessons are you taking away from the transition to remote teaching due to Covid-19? What changes will you make to your land-based teaching moving forward? Add your comments below!

In reply to Allison Titcomb.

Certainly, thinking of evaluation as a transdiscipline or the alpha discipline that Scriven describes is one reason why I think evaluation can inform research-practice partnerships. I think it’s also just the consultancy nature of what we do and the importance we place on interpersonal factors. like communication and relationship building.