drbethsnow

And the final reflective practice competency is:

1.8 Engages in professional networks and activities and contributes to the evaluation profession and its community of practice.

There are several ways that I engage in this competency:

• involvement with the Canadian Evaluation Society (CES)
• teaching evaluation
• mentoring my evaluation team
• social media involvement

Involvement with CES

I first got involved with CES back in 2010, when I was looking to find my way in this profession. The national conference was being held in Victoria, so I volunteered for the conference as I figured it would be a good way to meet other evaluators and learn about the field. And was I ever right – the evaluation community was so welcoming and I met people there that I’m happy to call friends and colleagues to this day.

For the next several years, I went to the CES national conference when I was able to attend, but then in 2015 the BC & Yukon chapter decided to host a one-day conference of its own, and that’s when my involvement really took off. I volunteered to be the conference program chair for that conference – and also volunteered to be a program co-chair for the national conference which was scheduled to be held in Vancouver in 2017. That role was a tonne of work, but it was also a lot of fun, as I got to work with two delightful fellow evaluators, Sandra Sellick and Wendy Rowe. I really enjoy and get a lot from conferences (both in the content I learn and in the networking opportunities they provide) and I know from experience that they take a lot of effort, so I think that volunteering for conferences is an important way that I can contribute to the profession and its community of practice.

Also in 2015, I joined the CESBCY council as a member at large, later transitioning into the VP role when the VP stepped down. In 2017, I became the chapter president. I’m really proud of the work the chapter is doing – we are hosting a lot of professional development events (e.g., one day conference, various workshops and webinars) and meetups that serve the evaluation community.

This year I also coached a student case competition team at the CES national conference – and that was a really rewarding way to support new evaluators in our community!

Teaching

Another way that I feel that I contribute to the evaluation profession is by teaching evaluation. I’ve taught evaluation courses at both SFU and UBC, and I’ve supervised practicum students from SFU, UBC, and UVic. And several of my students have gone on to work in evaluation (right now, I have three of my former practicum students and two of my previous evaluation course students working as evaluators on my team!)

Mentoring

And speaking of my team, I currently have 10 evaluation specialists working on my team and a big part of the work that I do as the leader of the team is to mentor and support them. This is another way that I am working to contribute to the future of our profession.

Social Media

Another way that I’m involved in evaluation professional networks is online. There’s the #EvalTwitter hashtag that a lot of us connect through. There’s even a monthly #EvalTwitter tweetup on the last Thursday of every month (at 5:30-6:30 pm Pacific time). And through#EvalTwitter I learned about Eval Central, an online forum that “aim[s] to encourage positive and fruitful discussion among culturally diverse evaluators from around the globe.” So I recently joined that and am eager to see what kind of conversations happen there.

Image credits:

• CES logo is from https://evaluationcanada.ca/
• Social media icons image posted on Flickr by Sean MacEntee with a Creative Commons license.

I didn’t write a blog posting in my series last Sunday – the weekend was busy and time got away from me! But it’s now this Sunday night and I’ve got cup of tea and I’m ready to reflect on reflective thinking!

1.7 Uses self-awareness and reflective thinking to continually improve practice.

Often I do my my reflective thinking over a a cup of tea – whether sitting on my own to do some reflective writing, or chatting with colleagues (As an aside, if you want to read some brilliant thoughts on reflective practice, check out Carolyn Camman’s fabulously titled blog posting “The coffee is largely metaphorical“). I’m an external processor and I find that I tend to come up with a lot of my great “a ha!” moments when I write my thoughts down or talk to a friend or colleague. I also don’t have a great memory, so when I have an insight, I need to write it down to cement it in my brain (or the very least, so I can look it up again later.)

Journalling

I write a lot of reflections as I go about my work. Whether I’m collecting data, analyzing data, in a meeting, or whatever activity I might be doing, if I have an “a ha!” moment, I write it in my reflective journal (which for the project I’m currently working on is typed up and saved on a shared drive with the rest of my team’s reflections, as these “a ha!” moments are about the content of the evaluation that we are working on together). A reflection might be about a pattern I’m noticing in the data, or a connection I’m making between different parts of the evaluation, or a surprise that I wasn’t expecting, or thoughts on some of our longer-term evaluation questions. My general rule is “if it’s interesting enough for me to want to tell my team about this cool thing I saw or thought of, it should write it down as a reflection). This improves my practice because it helps me to identify things that are important to the interpretation of the data, which allows me to develop accurate and comprehensive evaluation findings.

I also keep some separate reflections that are more for myself than as part of the evaluation data. For example, since I’m the team manager, if I have reflections that are about my work as a manager, and I might not want to share those with the team right away – especially if I’m trying to work through a challenge or figure out a way to be a better team manager. Some of those reflections might become things that I do want to talk about with my team later, but sometimes I need some time and space to work through stuff first. This helps improve my practice because being an effective leader will help my team be effective in its work.

Team reflections

Speaking of my team, we’ve taken to having group reflection sessions after we complete any big chunk of work where we debrief on:

• what worked well
• what didn’t work well
• how might we have done things better
• what can we glean from what worked well/didn’t work well to improve our practice for our next task

These are some pretty standard evaluation type questions, but we’ve definitely been able to continually improve our practice by doing this reflection together.

For example, in our first big round of data collection, we didn’t do nearly enough documentation of our data analysis. And with having a big team of people all working on different pieces of the data analysis, it meant that we had a lot of files that we’d all named in different ways, with our spreadsheets set up in different ways and often not very well labelled. So when it came time to write up our findings, it was quite difficult to find the data we needed, and we sometimes had to reproduce some of the analyses to ensure we had the correct data. So my big lessons learned for future rounds of data collection were:

• we needed standardized naming convention that we all used
• we needed all steps of analysis clearly documented so that another person could pick up the file and understand exactly what was done (without having to sift through formulas and pivot tables to figure out what it all meant)

These seem like pretty basic things – and they are – but this was the first time for all of us working on a big team. We each had our own individual naming conventions and ways of setting up our analyses in our spreadsheets that had served us well working as individual and what we hadn’t realized was how many different ways people could do the same task! Since the project is being implemented in a phased approached, we are now entering a period of time where our work will be a bit cyclical (collect baseline data for a site, monitoring data at the time of implementation, collect post-implementation data 3-6 months later, and repeat for the next site). And I can see that we are getting better and better each time because we’ve been reflecting on how we do our work and finding ways to be more efficient and more effective.

Another reflection that I shared in a team reflection session recently was something that I think links to the “self-awareness” part of the competency. Working in healthcare, even as a non-clinician, you get exposed to situations and information that can be quite emotional. For example, even when doing a chart audit, you get exposed to stories of serious illnesses/injuries and deaths. Or when interviewing healthcare workers who are exposed to traumatic situations, you also get exposed to those traumatic situations. As human beings, this can bring stuff up for us (like similar illness, injuries, patient journeys, and deaths of loved ones, for example) and it’s important to be kind to ourselves when stuff like this gets to us. I am extremely lucky that I work in a large team made up of kind and caring colleagues, so we know that we can go to each other if we need to debrief, or if today is just not a good day for us to do that particular observation or interview. Being aware of situations that might bring up things for me and being aware of my emotions as I’m experiencing them can help me to manage those, ask for help when I need it, and thus help to ensure that they don’t negatively effect the work. It can also help me to be empathetic to my colleagues and the people I interact with as I do my work.

In addition to reflection with my team of evaluators at work, I am also part of a co-op inquiry group that meets monthly to reflect on a particular topic (for us, it’s “boundaries in evaluation”) and that has been an amazing experience to hear the reflections of a group of evaluators from different sectors and locations – I have left every meeting having expanded on ideas I’ve been having and having learned new ideas or perspectives from my colleagues that have resonated with me.

Teaching

Teaching is a fantastic opportunity to reflect. Whenever I prepare to teach an evaluation course, I’m dedicating time to stepping back and thinking about the big picture of evaluation – what it is and how to do it well. I find it also brings me back to the basics and it gives me the opportunity to think about whether there are ways that can improve what I’m doing. I use a lot of storytelling and examples when I teach – I’ve had many students tell me that they really appreciate that I do that because I tell them “what really happens, as opposed to what the textbooks tell you it’s going to be like”. But it also helps me because, again, it gives me an opportunity to think about how I’ve done my work, how it links to concepts, theories, standards, etc. and how I might do my work in the future.

In addition to getting back to basics, I also like to tell students about whatever the “hot topics” are in the field at the time, which means that I have to keep abreast of what the hot topics are, and typically do a bit of research to be well versed enough in the topic to discuss it with the class. This is an opportunity for me to identify gaps in my knowledge and do some learning.

Another aspect of teaching that I think is reflective is that students tend to ask really great questions. And since they are coming from a different perspective, sometimes those questions are things that I haven’t thought about before, which forces to me to reflect situations from a different angle. Sometimes they ask questions that I do not know the answer to – when that happens, I tell them that I’ll go do some research and get back to them. This links to that notion of self-awareness – knowing the limits of my knowledge, having the confidence to say “I don’t know that right now, but I will find out”.

Blogging

And finally, this blog is something that I’m using as part of my reflective practice now. I’m glad that I decided to write this blog series on the evaluator competencies as a way to provide some structure and timeline to get me in the habit of reflecting here on a regular basis ¹Last Sunday notwithstanding.. I’m finding it quite useful to spend a bit reflecting on the extend to which I have each of the competencies and areas for each where I can continue to learn and grow.

Image Source:

• Pot of tea photo posted on Flickr by Jack with a Creative Commons license.

1.6 Is committed to transparency in all aspects of the evaluation.

transparent adjective
trans·​par·​ent 

a: free from pretense or deceit : FRANK

b: easily detected or seen through : OBVIOUS

c: readily understood

d: characterized by visibility or accessibility of information especially concerning business practices

Merriam-Webster Dictionary

To reflect on this competency, I decided to see what the definition of “transparency” is. I usually think of transparency in the sense of “sharing all the information”, which is a bit more extreme than one can actually be in an evaluation. For example, we have an ethical responsibility to maintain confidentiality for participants in our evaluations when they want their identity to be kept confidential. Sometimes we are working with proprietary information that the organization requires to be kept confidential. So as with so many things, being “transparent” requires a bit of nuanced thinking.

I used to work with someone who talked about her role in a communication chain in a hierarchical organization, where information came from the top and was cascaded down through the org chart. Sometimes, information was only allowed to be shared to a certain level – say, it could go from the VPs to the EDs to the Directors, but the Directors were not allowed to share it with the Managers – at least not yet. And this person’s (who was in a Director role) approach to it was to tell their managers “I do know this information but I am not allowed to share it with you at this time.” And then they would give the reason (e.g., “Leadership is planning to do X, but until it is signed off by the board of directors, it’s not official and so they do not want put this information out broadly in case the board does not sign off it on, as it could cause confusion.”). And then they would make a commitment to tell their managers as soon as they were allowed to. This approach stuck with me, because it was honest (they weren’t saying “I don’t know this information” when they really did know, which is an approach I’d seen others take in these types of situations) and it was as informative as it was possible to be given the situation – giving a reason why they weren’t allowed to share the information at that time, rather than just saying “I’m not allowed to say”. I find that not giving a reason usually results in people coming up with their own theories about what information is being kept hidden – and that ends up causing rumours and confusion. So I think that this (sharing what you can and being honest about what you can’t share and why) can be a useful approach to being transparent. Of course, there can be good reasons or bad reasons for not wanting to share information, so I think we also have a responsibility to think critically about the reasons why an organization might not want to share and to push back in situations where appropriate (e.g., if an organization wants to suppress evaluation findings because they think it makes them look bad, as I talked about last week, I’d push back on that).

I was interested to see that the definition of “transparent” isn’t just about making information accessible, but also making information “readily understood” and “free from pretence or deceit”. These are things I can get behind. Obviously, a credible evaluation should not include anything deceitful, but I think making information “readily understood” is something that is sometimes overlooked. There are so many ways that we can exclude people from evaluation by not being “readily understood” – whether that be by the way we design our evaluations, the ways we recruit participants, the methods we use, or the way the report the findings. There seems to be a lot of interest in the evaluation world around data visualization – i.e., presenting data in ways that actually convey the meaning of them. This is something that my team and I are actively working to get better at. And there’s interest in alternative reporting formats – i.e., not just handing over a 200 page report, but actually thinking about ways to report evaluation findings that work for those who are interested in those findings.

Something I see spoken about less often, but that I think about a lot, is the use of language. I’m a bit fan of clear, simple language when writing ¹Though I will admit that I have a tendency to be wordy, I write complicated sentences, and I overuse footnotes to an excessive degree. because I think that if I’m writing something, I want people to understand it. I mean, isn’t that why I’m writing it? I try to avoid jargon (or at the very least explain any jargon that I use) and prefer to pick a simple word over an obscure one. But I often see writing that is full of jargon, and unnecessarily large and obscure words. Part of me thinks that people write this way in an attempt to look intelligent. And I have seen situations where people use jargon as a way to try to cover up that they don’t know what they are talking about (which becomes evident as soon as you start asking questions like “What do you mean when you use the word X?”) An even more cynical part of me thinks that people write like this in order to exclude other people, by making the knowledge they are ostensibly trying to “share” non-understandable by “others” who don’t have the same training/background as them. After all, knowledge is power and keeping knowledge away from others by making it not understandable to others is a way of holding onto power. Which to me, is another reason to make the effort to make my writing as clear and easy to understand as possible.

At any rate, I hadn’t really thought about making information “readily accessible” as being part of “transparency” before, but it makes sense when I think about it.

Image Sources:

• Orange glass winged butterfly posted on Flickr by Alias 0591 with a Creative Commons license.
• Black glass winged butterfly posted on Flickr by Mary Shattock with a Creative Commons license.

My notes from the third and final part of the American Evaluation Association (AEA) eStudy course being facilitated by Jonny Morell.

• Jonny answered my question from last week:what is the difference between conceptual use and metaphorical use
  • there are fuzzy boundaries
  • if you think about chaotic systems – “strange attractors” (a.k.a., chaotic attractors)
  • you can do the math to plot a fractal – that is a technical meaning of the word
  • a conceptual meaning of the word – I know it’s not random, but I can’t tell you from one instance to the next where it will be, but I can tell you where it won’t be. You aren’t using the mathematics, but you are using the concept. Conceptual is still grounded in the technical.
  • metaphorical use – a step further away – we have this concept of chaos – that means its unpredictable. Conceptual means you have to “stay close to the mathematical, without doing the math”.

• he thinks that if you take complex behavoiur seriously, you’ll do better program design and evaluation
• but not trying to convert everyone to complexity thinking for everything all the time

Unintended Consequences

• he tends to think that unintended consequences are usually negative – any change perturbs a system, and even if some parts of a system aren’t working, it will mess up a system; it’s harder to make things work than to make things not work, so if you perturb a system, it’s more likely that bad things will come out of it
  • he’s heard this from many people with “broad and deep experience” whose work he respects
  • “Programs like to optimize highly correlated outcomes within a system. This is likely to result in problems as other parts of the system adapt to change.
  • Change perturbs systems. Functioning systems require many parts of “fit”, but only a few to cause dysfunction”
• he recently read about some work that shows this might not be true! But he wants to read more about it.
• there are always unintended consequences – and if they are good or bad is an important question!

• examples of unintended consequences provided by an audience member. A medical school started at a northern university to promote more physicians to work in the north, but saw unanticipated consequences:
  • positive: changes in the community (e.g., more rentals, excitement in the community about the work being done at the university, culture of the community changed: a symphony was started in the community)
  • negative: other programs felt snubbed

• Jonny wrote a book a while ago about how to evaluate unintended consequences (Evaluation in the Face of Uncertainty: Anticipating Surprise and Responding to the Inevitable)

Small Changes

• “because of sensitive dependence, it may be impossible to specify an outcome chain”
• e.g., sometimes programs evolve because of small things – e.g., because the program had time to do something that wasn’t in the original scope, or because the board agreed that something that wasn’t originally in scope still fit within the mandate

Unpredictability

A neat example of how difficult it is to predict the future is shown in this letter from Rumsfeld to G.W. Bush.

• “the commonly accepted view of logic models and program theory may be less and less correct as time goes on”
• there is debate over whether there are “degrees of complexity” (or if something is either complex or it is not”
• some think that even if you start with a simple system that can be reasonably represented by a logic model, over time it will transition to complexity behaviour (he doesn’t believe there are “degrees” of complexity, so it’s not that a simple system smoothly transitions to a complex one

Network Effects Among Programs

• imagine you have one universe where:
  • two programs: one on malaria prevention and another one that is promoting girls education –> increased civic skills
• and another universe where:
  • you have those two programs, but also other programs with goals around crop yields and road building = and all the programs interact with each other. E.g., if people are healthier (no malaria) and well fed (crop yield), you can work harder and increase economic development, which can feed back into the other programs, etc.
  • he thinks that this interconnected universe can have bigger effects over time
  • effectiveness can build over time with networked programs (whereas non-networked programs would just have the effect of the program and that’s it)
• challenge: how do you evaluate this when programs (and evaluations) are generally funded for single programs (or at least within a single organization), but not across multiple programs in different areas
• but there can be some programs that can spur change in all kinds of other areas of the system (e.g., ensuring everyone has a base level of education could –> increased civic engagement, increased health, increased economic development, etc.)

Joint Optimization of Unrelated Outcomes

• e.g., a program to try to decrease incidence and prevalence of HIV/AIDS
  • increase service –> decrease incidence and prevalence of HIV/AIDS
  • increase quality of service –> decrease incidence and prevalence of HIV/AIDS
  • decrease incidence and prevalence of HIV/AIDS –> better qualitity of life
• this is a fine program model
• all these outcomes are correlated
• you pour a lot of money into this program – lots of people make career choices, intellectual capital goes there
• so what happens to other things in the system?
• less people, money, etc. to go to women’s health, other health services
• so perhaps we see improvements in HIV/AIDS outcomes, but then you see worse outcomes in other areas of health
• so instead of doing that, let’s jointly optimize unrelated outcomes
  • e.g., instead of trying to optimize just HIV/AIDS outcomes, but try to optimize health overall
  • of course, this is hard to convince people of this – how do you decide how much each different group gets
• another example, you can drill people on reading to get them to do well on a test, but what if that makes them hate reading? Try to optimize that they do well enough on reading but also love reading

• have you ever seen HUGE logic models – lots of elements and lots of arrows?
• when you look at these, do you really think they are going to be correct? there’s lots of stuff that we don’t really know for sure; there are feedback loops that may or may not be true (feedback loops do tend to
• famous picture of dealing with insurgent situation in Afghanistan – you look at it and think that it can’t possible be right on its whole – things like sensitive dependence, emergence, non-linear effects of feedback loops, etc., etc. aren’t accounted for here
• it’s OK to have these big complex models, but it’s not OK to think that the whole model is true (even if you have data on every arrow within the model – because it doesn’t account for howcomplex systems behave). You can use the big model to look at pieces of it and think about how they relate to other parts of the model
• he has a blog posting on “a pitch for sparse models” – if things happen in the “input” and “activity” realm, things will happen in the outputs/outcomes realm
• he thinks that people can’t really specify the relationships in the level of detail that we usually see in big logic models (and he thinks it’s egotistical to think that we can do that).
• but it’s not very satisfying to stakeholders to say “we can’t tell you anything about intermediate outcomes”
• evaluators are complicit – we make these big models and stakeholders like it (and he says he is as guilty as anyone else at doing this)

Attractors

• if you push something out of place and there is an attractor present, it will go back
• e.g., rain that falls all ends up in the river, push a pendulum and ultimately it will end up back in the middle, planetary motion – gravity holds planets in their orbits, kids like playgrounds – kids will end up there, animals go to the waterhole
• “explains why causal paths can vary but outcomes remain constant”
• attractors are useful because:
  • lets you conceptualize change in terms of shape and stability
  • insight about program behaviour outside of stakeholder beliefs
  • promotes technological perspective: what will happen, not why

How do you decide if you should use complexity thinking in a given evaluation?

• more work to incorporate complexity into an evaluation (than, for example, basing an evaluation on a simple logic model)
• the evaluator – and the evaluation customer – should think about whether the value that is added by doing so is worth the extra work

For Further Reading

Jonny provided an extensive reading list. Here are some that caught my eye and I’m planning to check out:

• Gates, E. F. (2016). Making sense of the emerging conversation in evaluation about systems thinking and complexity science. Evaluation and Program Planning, 59, 62-73 (PubMed)
• Lawlor, J. A., & McGirr, S. (2017). Agent-based modeling as a tool for program design and evaluation. Evaluation and Program Planning. 65:131-138 (PubMed)
• Morell, J. A. (2010). Evaluation in the Face of Uncertainty: Anticipating Surprise and Responding to the Inevitable New York: Guilford.
• Morell, J. A. (2019). Revealing Implicit Assumptions: Why, Where, and How? https://www.crs.org/sites/default/files/report_revealing_assumptions.pdf
• Walton, M. (2016). Expert views on applying complexity theory in evaluation: Opportunities and barriers. Evaluation, (Sage)
• Williams, B., & Imam, I. (2007). Systems Concepts in Evaluation. Point Reyes, CA: EdgePress of Inverness. (online pdf)

Image Sources

• Blue ropes photo posted on Flickr by Joe Lodge with a Creative Commons license.
• Green and yellow tubes photo posted on Flickr by alex de carvalho with a Creative Commons license.
• Spiky thing photo posted on Flickr by Manel Torralba with a Creative Commons license.