allblogs

Nov 18 2019

Visual Thinking 101: How to Use Hand-Drawn Visuals to Increase Clarity and Communication of Evaluation Concepts

Hi #Eval19 friends! I’ve just returned home from the American Evaluation Association Annual Conference in Minneapolis. In addition to defrosting, I am also reflecting a lot on my experience at AEA this year. So many of my favorite moments at the conference this year happened in the half-day professional development workshop I led on visual note-taking. I was really inspired by the amount of courage and appreciation in the room. I really liked how everyone had a growth mindset about the exercises and was appreciative of themselves and others. I came away from the experience feeling really uplifted and encouraged. I love training on visual note-taking and visual thinking in M&E! Take a look at the handbook and slides I created for the training (with some visuals pulled out below). Please feel free to get in touch with me if you are interested in pursuing 1-1 visual note-taking coaching or training on visual note-taking for your team or organization. Thanks everyone! Have a wonderful week.

Nov 01 2019

Systems Change: The 6 Levers

Oct 30 2019

CES Webinar Notes: Retrospective Pretest Survey

These are my rough notes from today’s CES webinar.

Speaker: Evan Poncelet

was asked “are retrospective post test (RPTs) legit?”, so it did some research on them

you can’t always do a pre-test (e.g., evaluator brought on after program has started; providing a crisis service, you can’t ask someone to do a pre-test first)
“response shift bias” – “you don’t know what you don’t know”. Respondents have a different understanding of the survey topic before and after an intervention. So they might rate their knowledge high before an intervention, then they learn more about the topic during the intervention and realize that they didn’t actually know as much as they thought they did. So afterwards, you rate your knowledge lower (or rate it as the same as before the intervention, but only because while you learned a lot of stuff, you also know more about the topics that you still don’t know). So you have a different internal standard before and after the intervention that you are judging yourself against.

a brief history of RPTs
- emerge in the literature in 1950s (not much research on them – more “if you can’t do pre/post, do RPT”)
- 1963 – suggested as an alternative to pre/post or a supplement (if you do both pre test and an RPT, you can detect historical effects)
- 1970s-80s – suggested as a supplement to pre-test; research on RPTs (as a way to detect response shift bias)
- now – typically used in place of pre-test; common in proD workshops (e.g., a one-day workshop)

what do they look like?
e.g., give a survey after a webinar:

	Now	Before the Webinar
I’m confident in designing RPT	Agree Neutral Disagree	Agree Neutral Disagree

But if you have the pre next to post on the same survey, very easy to give a socially desirable answer or to have answer affected by effort justification (i.e., people say there was an improvement to justify the time they spent taking part in the program)

give separate surveys for pre and post (to reduce the social desirability bias)

research shows that separate surveys does show reduced bias, more validity
another option: perceived change:

	Now	Rate your improvement attributable to webinar
Your confidence in designing RPT	Low Med High	None A little Some A lot

research shows this option shows this is subject to social desirable bias
not a lot of research (could probably use more research)

advantages of RPTs
- addresses response shift bias
- provides a baseline (e.g., if missing pre-data)
- research supports validity and reliability (e.g., an objective test of skill is compared with results of these surveys)
- can be anonymous (don’t have to match pre- and post-surveys via an ID)
- convenient and feasible

disadvantages of RPTs
- motivation biases (e.g., social desirability bias, effort justification bias, implicit theory of change (you expect a chance to happen, so you report a change has happened)
- can use a “lie scale” (e.g., include an item in your survey that has nothing to do with the intervention and see if people say they got better at that thing that wasn’t even in your intervention – detect people over inflating the effect of the workshop)
- memory recall (so be very specific in your questions – e.g., “since you began the program in September…”). If you have long interventions, may be really high to recall

program attrition – missing data from dropouts (could actively try to collect data from the dropouts)
methodological preferences of the audience (what will your audience consider credible. RPTs are not well known and some may not consider them a credible source)

Other Considerations

triangulate data with other methods and sources (a good general principle!)
do post-test first, followed by RPT (research shows this gives respondents an easier frame of reference – it’s easier to rate how they are now, and then think about before)
type of information being collected:
- if you want to see absolute change (frequency, occurrence) – do traditional pre/post test (it can be hard to remember specific counts of things later)
- changes in perception (emotions, opinions, perceived knowledge) – do RPT

Slides and recording from this webinar will be posted (accessible to CES members only) at https://evaluationcanada.ca/webinars)

Oct 17 2019

A Failure to Plan….You Know the Rest of the Story

A failure to plan……

“A goal without a plan is just a wish.” ― Antoine de Saint-Exupéry, writer and pioneering aviator

Two things people who know me will tell you is that I love to cook and I love to travel. I am purposefully working through my bucket list and it is very long list!

For a long time, Alaska has been one of the places I most wanted to see. So, this past summer, I finally made it happen. My family and I spent two weeks driving the interior of Alaska. The trip involved 23 hours of driving, two small plane flights, 3 national parks, a ferry ride, and a hike on a glacier. I planned this trip for about 8 months. I read about the “Last Frontier,” I spoke with friends who had made the trip, followed bloggers and spent hours researching on the internet.

We had an amazing adventure. We ate some of the freshest seafood I have ever had –and that is saying a lot since I grew up in Florida. We saw wildlife and glacier after glacier. We met some amazing people that I don’t think I will ever forget.

And then this happened…….
I promise I did not use a filter on this picture. This gloomy shot was the result of one of the many fires that burned in Alaska this summer. Record temperatures and dry conditions created fires later in the season than ever before, damaging thousands of acres, destroying property and contributing to air pollution. In Alaska, climate change is a reality, not just a political argument.

For my family, the impact was a disappointing inconvenience. It meant a 3-hour wait in traffic on Sterling Highway only to be turned around when the Swan Lake fire jumped the road. We were forced to regroup and punt since the Homer leg of our trip was just not going to happen. We made the decision to head back to Anchorage. Unbeknownst to us, there was another fire between Denali and Fairbanks, causing smoke to hang over Anchorage for the rest of our trip.

But life is like that right? We make a plan, we do the research, invest the time, but things just don’t turn out the way we want.

In community work, things like this happen all the time. Its hard to get a group of community members, all representing different organizations with different agendas, to envision a common purpose for their community. It is just like herding cats.

That’s why I find logic models so darn helpful. They may be despised by some, but I believe they are despised because they are oftentimes overly complicated. (I certainly have been guilty of creating a few that were way too complicated myself). But I have experienced over and over again a situation in which the program staff and leaders just knew they could explain their program clearly. Until we went through a logic model process, and they couldn’t.

Lately, I hear more nonprofit leaders, coalitions, and collaboratives sighing over the thought of creating yet another logic model. If this is the case, a tearless logic model is a great way to develop a logic model without ever mentioning the words “logic model,” “activities” or “outcomes.”

Another reason logic models are, well lets just go with “not held in high esteem,” is because many nonprofit or community leaders just do not know what to do with it once they have one. Short of using it as training paper for your pup….here is why you might need one and how you should use it.

This list is in honor of the community coordinator who recently asked me, “WHY DO I NEED A LOGIC MODEL” &%$%$!!!!!!.

A logic model helps:

Serve as an organizational development tool. It helps your organization define its vision and goals, gets everyone on the same page, and everyone understand where they fit in the work.
Explain to others (funders, community members etc.,) in a visual way, what you want to accomplish.
Make clear to you (and your board) what to say “Yes” or “No” to.
Clarify what is feasible and achievable.
Connect the dots so that your work is linked (activities lead to outcomes).
Define your evaluation questions.
Ensure you have what you need to be successful.
Test the “logic” of your program or strategy: Is what you are doing really going to result in the outcomes you propose?
Demonstrates your commitment to high quality work.
Serve as a program management tool that drives your workplan.
Satisfy a funder’s “request” for a logic model.
Guide your next step. It is your roadmap when you get lost.

Getting back to my Alaska trip, and I wish I could go back……Because I had done all of that research and had planned so well, when we were thrown that curveball, we could manage our next step. We got out the map, thought through our options, and made the best decision we could under the circumstances.

Working for social change is like that too. Your programs and strategies don’t always turn out the way you planned. You lose that critical funding you were counting on, staff leave, stakeholders come and go, local politics influence your program in not so great ways. Things happen!

Having a logic model is that guidepost you need in order to plan for change. It is the roadmap when things just don’t work out like you planned.

Ready to get started? I promise, no tears.

Oct 11 2019

Adventures in Teaching: Exam Post-Mortem

I have been crazy busy with my first semester as an assistant professor of psychology in evaluation at University of Wisconsin-Stout, but I wanted to share with you an activity that I found very beneficial after our first exam in Introduction to Psychology. I have heard these called multiple things–post-mortem and wrappers are the two …

allblogs

Footer

Follow our Work