Date: 14 April 2022

Hosted by: AEA, Government Accountability Office

Speakers: Karen Kirkhart, Kathryn Newcomer, Giovanni Dazzo, Nicole Bowman, Terell Lasane (moderator)

This was a really great webinar. I furiously took notes of as many of the insightful things the panelists were talking about. The notes are imperfect (I tried to catch some direct quotes inside quotation marks, but some of this is paraphrased – any errors are my own! If you are really interested in this topic, you can check out a recording of the webinar here and there are a bunch of resources that the speakers shared at the end of this posting.

Karen Kirkhart:

• multicultural validity is a “call to broaden the kinds of evidence that are considered in validity conversations”
• limited views of “validity” promotes social injustice – it silences
• 5 intersections sources of intersecting validity evidence
  1. methodological validity – the stuff we usually think of re: quant and qual (insufficient as the only source of evidence)
  2. theoretical evidence – “insights from social sciences and humanities and professions”, Indigenous wisdom; program theories (examine these for bias towards deficits and disadvantage)
  3. relational evidence – “how people relate to one another, to our planet, to the universe”; “how power is exercised in relationships”; collaborative and participatory approaches ‘ position relationships as positive, but this is not always true. e.g., “inclusion” can “twist” into a settler invitation to assimilate
  4. experiential evidence grounds our understanding in the lives of the community members; “calls evaluators to spend time with communities, upon being invited”
  5. consequential evidence – brings accountability to our work; examine what happens or fails to happen as a result” if “evaluation does not move the needle towards social justice, what does that tell us about our accuracy and adequacy of our prior understandings?”

Kathryn Newcomer:

• multicultural validity is a lens through which we should view our claims (e.g., claims of
• “evidence-based policy making” has been embraced in OECD trails and a focus on RCTs as being the way to demonstrate evidence
• likes the term “impactees” rather than “beneficiaries” (because you don’t know if they are benefiting!)
• concerned with standards used in various registries to judge research
• working on an advanced set of evidence standards – broadening view of causation, context, equity
• fit methods to the questions
• 3 books influential in her thinking of cultural humility and in understanding racism, sexism, and classism (Stamped from the Beginning by Ibram X. Kendi, White Trash: 400 Years of Untold History of Class in America by Nancy Isenberg; Invisible Women: Data Bias in a World Designed for Men by Caroline Criado Perez)

Giovanni Dazzo

• evaluations often based on the opinions of what is “rigourous” according to the funder and the evaluator, but not necessarily on the people who the program is supposed to serve
• we as evaluators often term sticky note activities as “participatory”, but is that what the community consider to be the ways they participate
• if we enact oppressive ways of “participating”, we are robbing people of their identities
• how can our practices restore our humanity as evaluators?
• “an expertise that privileges distance (another word for “objectivity”)”
• co-constructed a reflective framework
• “the extractive nature of inquiry” vs. a way to restore
• “restorative validity”
• seek to heal and restore rather than to “prove a point”

Nicole Bowman

• storytelling as valid and impactful
• scientific and policy and academic humility to add to the idea of cultural humility
• we must understanding history in our context to walk together in a good way
• our experiences matter, how we got here today
• Braiding Sweetgrass by Robin Wall Kimmerer
  • braiding requires tension – “tension is respected and expected”
  • the more tension in the knot, the stronger it is – intersectionalities
• think of the 575 tribes as a nation state
• refers to self as a “blue collar scholar”
• “we have to get upriver”
• not much has changed despite many years of reports, etc.
• lets bring some wisdom into that work, instead of just “evidence”
• learn about sovereign nations
• build capacity, competency, and skills on how to work nation-to-nation
• how do you make RFP policies so that we can build things differently and start piloting and testing things to look for better outcomes
• who owns the data – how we publish
• if we are trying to learn how to do things differently together, we need to dedicate more time and resources to do that
• think about who is here and who is not here

Q&A

• KK:
  • There is not one “evaluation community” – only a small proportion of those are members of evolution associations
  • much evaluation is done by outside contractors
  • social impact investing” not part of evaluation community, do a lot of evaluation work
  • lots of people have not had training in evaluation, let alone training in culturally responsive evaluation, cultural humility
  • some foundations (like Kellogg) and organizations like CREA that have been doing this stuff for a long time
  • the Urban Institute – lots of free materials you can download
  • cultural humility is so important – you can never fully understand another community/culture, you don’t just do a training on cultural humility/responsiveness and say you are done
• KK
  • cultural competence is a stance – it’s infused across the AEA competencies, not a single “competency”
  • cultural competence implies an “end point” – that term may have outliving its usefulness
• NB
  • legal political aspects – Tribal Nations are the only groups within “cultural responsive” that have this status
• KK
  • there’s been work on cultural responsive evaluation for a long time (e.g., growth of TIGs, diversity work in AEA)
  • intersectionality theory has had a huge impact – “it messes everything up. which is a good thing”
  • things that disrupt and shakes us up is a
  • within society at large, the pandemic has raised awareness of inequities and the anger and outrage of the murder of Black citizens
  • and recognition that historic “solutions” have not been working
• TL
  • if you codify things into law, it changes society
  • The “evidence act”
  • current administration released something talking about the importance of Indigenous wisdom
• NB
• younger generations do not see disciplinary and other lanes, “everything is related”; they don’t see boundaries they see opportunities, “putting together this beautiful quilt”
• e.g., government TIG reaches out to Indigenous TIG all the time
• we need to braid this together
• TL:
  • I teach and our discussions show that students are thinking critically about how evaluations have not met the mandate because they are not considering cultural
• KN:
  • qualitative and mixed methods are more and more becoming the body of research and evaluation, we may have reached a tipping point
  • many of the standards of evidence are “canonized” with positivist notions of “validity”, but more qualitative researchers are coming to the fore to challenge this; KN’s new standards are in a manuscript she’s
• GD:
  • “we are more concerned with being ‘scientific enough’ than te are about being relevant”
  • demonstrated to where the money flows – to quant research – so those researchers hold more power and control
  • in participatory, community-based, it is assumed that “participation is good” , but as KK mentioned, it’s not always so
  • we have to ask why people are being asked to participate, are they being compensated? do they have time? often funders give excuses as to why “we can’t pay individuals”
  • processes often silence minoritized or under-resources communities
  • people often showcase the “participatory method” as the end goal, as opposed to how the method promotes mutual understanding, without that we don’t get to relational evidence, liberation
• NB
  • there’s an Indigenous data sovereignty network
  • they are publishing in the data science literature too
  • data = power
  • “I need courageous, compassionate, and curious people”
  • mostly white males and females fill these positions that have the power and priveldge
  • we have to talk about power and privilege and capitalism, uncomfortable things
  • red, white, yellow, black are all the colours on our medicine wheel, all working together
  • we have no business making policy on things we know nothing about
  • we are all learning different things – e.g., “I don’t have experience in LGBT+, but have been invited into the work because I know Native stuff and they know that I will come in a humble way”
  • you can learn about communities based on what they are posting in social media
  • we learn, unlearn, and relearn together
• GD
  • we have to think of where the money is going
  • evaluation work is contract based
  • we’ve broadened our thinking about how we do evaluation funding. Learning about how communities do things rather than funding projects for evaluators to go in and say “tell us everything you know”
• KN
  • book on inclusive engagement
  • we think “engagement” is saying “we are having a meeting on Wed at 7 pm so we can tell you what we are going to do to you” – that’s not engagement
  • what are communities getting from this?
  • need to think of inclusion at design stage, not just at evaluator
  • evaluators come into projects too late to do a lot of this work sometimes
  • the term “rigour” is interesting – has a specific ontological assumption that there is a truth that evaluations have to find; probably not how to think about it. tends to compete with ideas of multicultural sensitivities. A very rigid view of rigour
• KK
  • rigiour is often invoked against multicultural sensitivities. – my answer is that nothing is more rigiour than triangulating multiple sources of data
• NB
  • if your “rigour” is working why are Native people still experiencing such high levels of ditabets, suicide, lower rates of graduation at high school, universities – your rigour is not working, since we are not getting the outcomes
• “lets go beyond “do not harm” and be a good relative”
• TL:
  • when you get a “significant” result in an evaluation saying there’s a , people often don’t ask “does it work well for everyone? does it work well in different contexts”
• DG:
  • there are courses on decolonizing methodologies
  • where is the money going?

Resources

There were a tonne of resources suggested during the workshop. Here are some that I’m planning on checking out:

• Center for Culturally Responsive Evaluation and Assessment
• Braiding Sweetgrass by Robin Wall Kimmerer
• Red Earth, White Lies by Vine Deloria, Jr
• White Privilege and the Decolonization Work Needed in Evaluation to Support Indigenous Sovereignty and Self-Determination by Kate McKegg
• Bowman, N.R. (2020). Nation-to-Nation in Evaluation: Utilizing an Indigenous Evaluation Model to Frame Systems and Government Evaluation: https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20411 
• Bowman, N.R. (2018).  Looking Backward but Moving Forward: Honoring the Sacred and Asserting the Sovereign in Indigenous Evaluation: https://journals.sagepub.com/doi/abs/10.1177/1098214018790412. 
• Russo Carroll, etal. (2020). The CARE Principles for Indigenous Data Governance: https://datascience.codata.org/articles/10.5334/dsj-2020-043/. 
• SAMHSA: Native American Center for Excellence (n.d.): Steps for Conducting Research and Evaluation in Native Communities: https://www.samhsa.gov/sites/default/files/nace-steps-conducting-research-evaluation-native-communities.pdf
• Mariella, Brown, and Carter (2009). Tribally Driven Participatory Research: https://digitalscholarship.unlv.edu/cgi/viewcontent.cgi?article=1058&context=jhdrp
• Indigenous Research Methodologies by Bagele Chilisa
• Johnston-Goodstar, K. (2012). Decolonizing evaluation: The necessity of evaluation advisory groups in Indigenous evaluation. New Directions for Evaluation, 136, 109-117.
• Smith, L. T. (2012). Decolonizing methodologies: Research and Indigenous peoples (2nd ed.). New York: Zed Books.
• Dean-Coffey, J., Casey, J., & Caldwell, L. D. (2014). Raising the bar – integrating cultural competence and cquity: Equitable evaluation. The Foundation Review, 6(2). https://doi.org/10.9707/1944-5660.1203
• Kirkhart, K. E. (2016). Equity, privilege and validity: Traveling companions or strange bedfellows? In S. I. Donaldson & R. Picciotto (Eds.), Evaluation for an equitable society (pp. 109-131). Greenwich, CT: Information Age.
• LaFrance, J., Nichols, R., & Kirkhart, K. E. (2012). Culture writes the script: On the centrality of context in Indigenous evaluation. In D. J. Rog, J. Fitzpatrick, & R. F. Conner (Eds.), Context: A framework for its influence on evaluation practice. New Directions for Evaluation, 135, 59-74.
• Kirkhart, K. E. (2010). Eyes on the prize: Multicultural validity and evaluation theory. American Journal of Evaluation, 31(3), 400-413.
• Johnson, E. C., Kirkhart, K. E., Madison, A. M., Noley, G. B.,  & Solano-Flores, G. (2008). The impact of narrow views of scientific rigor on evaluation practices for underrepresented groups. In N. L. Smith & P. Brandon (Eds.) Fundamental issues in evaluation. (pp. 197-218). New York: Guilford.
• Stamped from the Beginning by Ibram X. Kendi
• White Trash: 400 Years of Untold History of Class in America by Nancy Isenberg
• Invisible Women: Data Bias in a World Designed for Men by Caroline Criado Perez