cplysy

Sep 23 2020

Unwritten Outcomes

Evaluation as a profession is built around the rational. So what about the things that seemingly defy logic?

So this is sort of a cartoon illustrated philosophical brain dump week. I’ve been diving deep into my public library’s audio book collection and thinking a lot about rationality. If you don’t want to think, just skim through the cartoons. You’ve been warned.

I remember thinking a bit about evaluation when perusing some of Daniel Kahneman’s work on behavioral economics. Economics traditionally was based on this idea that people make rational decisions. Behavioral economics calls that into question and it led to a Nobel prize for the psychologist, Kahneman.

And Kahneman’s work can now be seen transcending fields. I mean it came up most recently when I was reading a book by hostage negotiator Chris Voss.

Everything we’ve previously been taught about negotiation is wrong: you are not rational; there is no such thing as ‘fair’; compromise is the worst thing you can do; the real art of negotiation lies in mastering the intricacies of No, not Yes.

Never Split the Difference by Chris Voss

I had been waiting for the idea to creep into the evaluation world. Like for someone to write a book titled behavioral evaluation. To start making claims that people are not rational so how do we expect to create accurate rational models of the activities of people?

His central message could not be more important, namely, that human reason left to its own devices is apt to engage in a number of fallacies and systematic errors, so if we want to make better decisions in our personal lives and as a society, we ought to be aware of these biases and seek workarounds.

Daniel Kahneman changed the way we think about thinking. But what do other thinkers think of him?

But lately I’ve been feeling like saying we are not rational is just a cop-out.

Of course we’re rational. We make rational decisions all the time. But my rational isn’t necessarily your rational.

My desired outcomes might be different, and who is to say my desired outcomes will even stay fixed. Have you ever lost a game on purpose? Taking it easy on someone to help build their confidence. Or maybe another player does something that annoys you. Does changing your desired outcome from “winning” to “making sure that one jerk loses” mean you are not rational?

Maybe, just maybe, it’s not a lack of logic we’re noticing. It’s just people who don’t do what we think they are going to do.

It’s not the work of white Nobel prize winners that we need to see further integrated into our field. Perhaps though, it should be the work of feminist thinkers like Audre Lorde.

Rationality is not unnecessary. It serves the chaos of knowledge. It serves feeling. It serves to get from this place to that place. But if you don’t honor those places, then the road is meaningless. Too often, that’s what happens with the worship of rationality and that circular, academic, analytic thinking. But ultimately, I don’t see feel/think as a dichotomy. I see them as a choice of ways and combinations.

Sister Outsider: Essays and Speeches by Audre Lorde

But of course it has already.

I find myself “discovering” works well known to many. Works that have already inspired countless, and will only continue adding to those numbers over time.

I soon recognized that there was a severe disconnect between the grantmaker’s success measures and the mission and/or approach to the work of many of the grantees. I wrestled with how to provide enough insight toward demonstrating some sense of accountability for the grantmakers while also creating space to use the evaluation findings to support grantee effectiveness (e.g., delivering technical assistance).

I discovered along the way that the root of this disconnect was not a grantee’s ability to be “successful”, but the varied definitions used for success. While struggling to figure out how to effectively address this observation, I recalled the words of Audre Lorde, an author whose writings I fondly admire and reflect on regularly. In other words, many grantees are trying to “crunch” their work into someone else’s definition of success, and in that process, their true efforts and impact go unnoticed.

The Day Audre Lorde Inspired Me to Reconsider the Definition of Success by Alison T. McMcNeil

So wait, where was I?

Oh yes, rationality.

I think this is the lesson that’s now stuck to my heart.

Desired outcomes are as infinite as our motivations. In a single person they can change quickly with new context, or gradually over time. In groups of people they are never fixed.

As evaluators, we need to seek out these motivations and desired outcomes, both written and unwritten. Because rational exists, and that rationality is essential to understanding how programs work or do not work.

It just might not be the rational represented by the official program theory of change.

We believe the one who has power. He is the one who gets to write the story. So when you study history you must ask yourself, Whose story am I missing? Whose voice was suppressed so that this voice could come forth? Once you have figured that out, you must find that story too. From there you get a clearer, yet still imperfect, picture.

Homegoing by Yaa Gyasi

Sep 22 2020

Ending Dusty Shelf Reports: Interview on James Pann’s YouTube Show

I recently had the opportunity to be a guest on James Pann’s YouTube show to talk about reports. James is an Associate Professor, program evaluator, and psychologist who specializes in evaluating health, human service, and educational programs.

Watch Our Conversation

Here’s a recap of our conversation:

My Introduction to Dusty Shelf Reports

I recently had my 7^th year anniversary of working for myself. I started out as an evaluator and I always loved the data viz side of my job. Some people love doing logic models or lit reviews, me I love poring through spreadsheets trying to figure out the pattern.

Early on in my career, I was throwing all my effort into a report, staying late, working on the weekends, doing whatever it took to do my best work. My boss sat me down and kindly said, “Ann, you’re a great team member and I’m so glad to have you. I’ve noticed you’re working really late on this evaluation report. It matters, sure, but I don’t want you to sacrifice your personal life for a dusty shelf report.”

I’d never heard that term before and I instantly felt crushed, but also thinking, “there’s got to a better way- this can’t be the standard”.

Reasons Evaluators Should Be Learning About Data Visualization and Reporting

If you’re working in academia or are a student in grad school students, then you absolutely need to write academic reports. But if you go outside of academia, there are many other workplace settings that have their own standards. The bar is being raised so much (in a good way) in thinking beyond the report. Can we also have interactive dashboards? Infographics? Amazing things are being done.

The tricky part is that none of us were trained for how you go from really technical reports in academic settings to non-technical reports, non-technical audiences and really quick turnaround times.

Resources Someone Just Starting Out in Evaluation (and Others) Can Use to Get Proficient in Data Visualization

For a student just getting started or if you’re new to the evaluation field, you can take one of my courses (hint, hint, nudge, nudge).

Soar Beyond the Dusty Shelf Report is our free mini-course where you’ll gain practical techniques that you can apply immediately to improve your visuals.

For reporting the best place to start would be to do a self-assessment. Ask yourself: what is the impact of your reports (if any).

There are four red flags to really look for with reports (or slideshows or dashboards, etc.). Look out for these:

No Response
- No response at all after you send it out.
Promise to Follow-Up Later
- You get a response that says, “Thanks, I’ll let you know if I have any questions.”
“Compliments”
- You get a response that says, “Thanks, we can tell that a really technical team worked on this report. It’s very detailed and thorough.”
Won’t Read It
- They ask for another format entirely like a slideshow, one-pager, etc.

Creating a Dataviz Wall of Fame

One thing I did for myself early one that was so helpful was to create a data viz wall of fame. I used to look for examples of great graphs, dashboards, infographics, etc. and would print them out and plaster them on the wall above my desk.

Nowadays, you could have a Pinterest board, folder on your computer or favorite them on Twitter. Start by actively collecting examples that you love but then push yourself to the next level by thinking, “What’s the rubric for this? Why is it good- is it the writing, the type, etc.”. Once you can identify why it’s great, you can start to replicate it.

There’s a lot of great podcasts out there too. The ones that come to mind are:

Data Viz Today with Alli Torban

Data Stories with Enrico Bertini and Moritz Stefaner

Data + Love with Zack Bowders

PolicyViz with Jon Schwabish

Techniques for Engaging Project Stakeholders in the Reporting Process

I co-authored an article on using Data Placemats, which is a technique you can use to engage project stakeholders in the reporting process.

I want you to step away from the idea of waiting until the end of the project to give the completed report/slideshow and hope that it gets used. Instead there’s a three-step process you should follow:

Develop data placemats with preliminary findings- traditional format.
- Make a data placemat. Use Word or PowerPoint and lay out one graph per finding.
Meet with stakeholders and have them explain their findings in their own words; you’re a facilitator, not a lecturer.
- Hold a meeting with the stakeholders where you pose discussion questions and then sit back and listen. Walk them through how to walk through the data and have some of the ‘lightbulb’ moments themselves.
Write the final report- storytelling graphs- using their interpretations.
- Take all that data you collected from them and their insights and put that into the report.

Easy Fixes for Reports

Here are the most common mistakes–and easy fixes–that I see.

Going Beyond the Report

You only have a report. A report can’t meet every audiences’ need. You need one product per audience.

Structure Your Report

Not starting with the “So What?” The most important information is buried in the report. Put the “So What’ on the first page and get your audience in the game right away. Don’t risk losing your audience.

Not following brand guidelines. Use your fonts and colors, especially think about using your recipients brand guidelines. It’ll take you about 10 minutes but shows that you’re willing to go the extra mile and it looks more professional.

Design Each Page

Not aiming for 1+ visual per page. I define visuals very broadly. You might use the logos of organizations, or include a timeline to show the numerous steps taken.

Forgetting to lower the reading level. The average American reading level is 6th through 8^th grade. Plug your text into a readability level checker and make sure your reading level matches your audience. Aim to be about two levels below your audience’s educational level. For example, if most of your audience members have completed Master’s degrees, then aim for two levels below that (high school).

How to Obtain Feedback on Your Work

When you blog and YouTube like I do where your work is front facing, people will tell you! I remember posting my first YouTube video and the first comment was from an anonymous person who wrote in all caps, saying they couldn’t hear anything over the clicking of my keyboard and said it was such terrible quality. I didn’t even know people used a microphone! I watched YouTube videos on mic and lighting comparisons to get to my setup now.

I did a webinar a week or two ago and my phone number is on my website (I forward my business phone number to my cell phone). This guy called and said, “I paid for your webinar you did yesterday and I have one problem with it”. I instantly went into flight or fight mode wondering what he was going to say. He went on to say how most presenters just read off a script but you didn’t, it was one of the best he’d seen. People will just call and will tell you very openly how they feel about your work so you have to have a poker face.

Masterminds

I’m also in a few masterminds with some other business owners where we act as each other’s sounding boards and brain trusts. I’ll throw out ideas to them and they’ll flat out tell me that it’s not a great idea. It’s really helpful to have a peer group that you trust.

Collect Feedback After Trainings and Courses

When I do trainings, I collect feedback forms or do an internet survey. In my larger projects, I build it in to have a debrief. They turn into problem solving for when hiccups may happen in the future. In shorter projects, I look for repeat clients or referrals. If I never hear from them again, that would worry me.

At the end of my courses, student receive a course survey. I also have behind the scene data for course students that let me know how much of a course they’ve completed, how often they’re logging in, if they’re watching all of a video or not and more. I can see which videos people are re-watching or even which sections they’re re-watching which tells me where they’re getting stuck.

Inspiration for Developing Blog Posts

I usually write blog posts on questions that workshop or course participants are asking. If everybody is struggling on how do I make sure my colors are accessible, that’s a great blog post. I also do a lot of before/after makeovers which help people see all the little steps that can make a big difference. I try to show the behind the scenes thought process so that people can then replicate it in their work.

More recently I’ve been doing interviews to try and get to know the people in data viz better and then I turn that interview into a blog post.

Finding a topic that you personally care about is key. I don’t think blogging is supposed to feel like work. You could monetize your blog but it would take a lot to make a full time living from a blog, but it’s mostly a hobby and supposed to be fun. I love the idea of thinking about what’s interesting to you and what do you want to explore more.

Book Recommendations

I have gifted all of my siblings-in-laws, Your Money or Your Life by Vicki Robin. She published it back in the 90s for the first time and just did a re-do a couple of years ago. It’s a financial book kind of, but it’s more about thinking about your life energy. You’re only alive for so long and you have a limited number of days, how can you make the best use of your time? It’s about trying to avoid just consuming things that give you temporary happiness, but thinking about how do you create for long-term, life satisfaction?

My husband I listened to the audio version during a long car drive and kept pausing it to talk about the insights we were having. She says to not throw away your life in a job that’s only decent, you need to aim for a great job and set up. Or maybe you should stay home and be a parent. This is what inspired us to have my husband quit his job ad have our family join me on the road while I traveled teaching. This book clearly changed our life trajectory.

Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones by James Clear is so actionable for everybody in every industry. I think everyone can learn something from it.

James: Steve Kotler’s stuff on flow like the Rise of Superman: Decoding the Science of Ultimate Human Performance. Also, Bernardo Castro has a philosophy book on idealism vs materialism. It’s not some academic nonsense, but it’s very practical. I think he’s one of the most important writers right now.

How to Structure Your Day/Week to Enhance Productivity

That’s a tricky one because I think there’s perception that Ann Emery must wake up at 5 am, run 10 miles, then make a green smoothie and work, work, work until midnight. She never sleeps and must be miserable, clearly. In reality, I get a lot done but I also rest and have fun. I don’t see those as opposing dichotomous topics at all.

I think you can have this wonderfully fulfilling personal life and achieve a lot professionally at the same time. I’m into minimalism in terms of thinking about what really matters in your work and personal life (which ideally are the same thing!). But really thinking about what matters and let go of everything else.

My Husband is a Stay-At-Home Dad

My husband let go of his very good salaried job with a pension and top-secret security clearance. We let that go so we no longer have to do things like both waking up with alarm clocks, rushing the kids off to daycare, both working and then figuring out who makes dinner, etc. We now don’t set alarm clocks and get enough rest each day.

We Bought a House with a Quiet Office

Now if I have to set an alarm, I think to myself, “My life is out of control! Why did I schedule something before 10 am?” We purposefully stopped fulltime travel due to COVID and so when house hunting in Florida, we were looking for a very specific home. We wanted a one level house with a bonus space I could use as an office. It’s my little retreat- I go to my office, get all my work done and then I go downstairs and go ‘home’.

Be Smart About Scheduling

I used Calendly for scheduling meetings and I’m only available from Mondays at noon until Thursday. I don’t book anything on Monday morning, that’s a terrible way to start the week! And I don’t boo anything on Fridays, because if my brain is tired, I’m not going to be any good on a Zoom call. Fridays are usually a day for myself to write a blog post or record videos for online courses. It feels really good to end the week like that, doing something creative for yourself.

Creator vs. Manager/Delegator

Identifying whether you’re a creator or more of a manager/delegator and making sure you’re not trying to go back and forth. Instead of trying to spend one hour writing a blog post (creating), then a 30-minute meeting about a project and then spend one more hour writing that blog post. I try to really separate it and have a half day of uninterrupted to do the actual, creative work. Then I try to book all my meetings, back to back at another time.

Morning, Afternoon or Late-Night Person

I’ve listened to whole podcast episodes about being careful to know if you’re a morning, afternoon or late-night person. I used to be a late-night person and would do my best work at 9 pm-midnight. But now I’m a morning person so making sure that my mornings are my uninterrupted time. I rarely look at emails and instead am building courses and writing. In the afternoon, I take calls. You really need 100% of your brain energy for the creative work.

James: I think it’s great to get good sleep, that’s probably the first and most important thing. Getting that honed in has been huge.

And then being okay with taking breaks during the course of the day and actually building it in. Making sure that you’re going through a cycle and not just powering through.

Learn More

Subscribe to James Pann’s YouTube show.

Your Turn

Comment below: Which reporting tips were new to you? Which tips are you going to put into practice? Have reporting tips of your own to share?

Sep 22 2020

Predicting Next Year’s Top Story

Foresight can involve complex data gathering, sensemaking, and design and also be something as simple as developing the headline for next year’s news.

This simple technique can get your organization started on futures thinking and provide a way to connect the present situation with trends that you are seeing in your industry and the world to strategy and aspirations. This technique gets you to imagine the headlines of the future (what people are writing or saying about your enterprise) and walking back from that or projecting forward to fill in the steps that led you to that outcome.

Setting the Stage

This is an activity that is best done as a group anywhere from 4 to 12 people and can be done in as little as an hour, although it can be done over a longer period of time in single or multiple sessions if you wish to go deeper into the assumptions and models for the future.

It’s important to frame the exercise by choosing whether you want headlines for the organization or a specific project or product. It doesn’t matter who these imaginary headlines are to be written by (e.g., journalists, industry professionals), however, it does help to imagine what context they are to be written (e.g., news media, business press, industry news outlets, professional associations, peers). Make whatever context you pick clear even if it is in multiple contexts.

Next, set a time horizon for the headlines sometime within the next 6 to 12 months.

Lastly, this is often used to frame positive outcomes. However, after you’ve determined what success looks like it is worth considering repeating the exercise at a later date (e.g., one week later) with the converse: focusing on headlines that report failures, disasters, or problems. This can help your team see threats as well as possibilities.

The materials you will need are pieces of paper (preferably sticky notes because they are easily portable and can be re-arranged) and a whiteboard or flipchart/newsprint sheets of paper and markers for a facilitator (who can be external or a member of the team) and the team/participants.

Activity

Individually, have participants brainstorm headlines they imagine for the time horizon you have set. Give them about 5-10 minutes and ask participants to strive for volume — lots of ideas — over quality.

As a group, post together (with stickies) or share the ideas that individuals have generated. This can be done by having individuals post up their sticky notes on a wall and then later organized or by doing successive round-robin reporting where everyone presents a single idea in as many rounds as there are ideas.

We suggest having the group vote on headlines that they like, elicit the strongest reactions (positive or negative), or are the most provocative. Aim for 3-5 headlines. With these headlines explore as a group some of the assumptions that are in place for this headline to come true. The aim is to answer the question: what would have to happen for this to become a real headline?

Why?

This activity helps you set and frame a goal for your organization, project or product. It can help elicit information about what kind of aspirations, assumptions, and ideas that your team has about what you are doing. It will also allow to identify what kind of relationships, resources, or facilitators are needed to get from where you are to where you wish to go over the time horizon you’ve picked.

If you do the negative case headline, this technique can help frame what kind of necessary activities are required for success and where they can possibly go wrong. it will allow you to identify threats and risks associated with what you’re planning to help account for that in your plan.

This simple technique is powerful and can be used in a single session, with multiple units, or as part of a planning exercise and the dividends are great. It’s fun, creative, and informative.

If you want to see more about what this can do, contact us and we’ll gladly help you set up a foresight scan and strategic plan for your project or organization based on this kind of futures thinking.

Sep 22 2020

Comment on How to formulate strong outputs by Julius Awullama Abimiku

I need more of monitoring and evaluation knowledge.

Sep 20 2020

Tips and Tricks for Great Survey Design … and a Survey for You

We have all taken TONS of surveys in our lifetime.

We get surveys when we make an online purchase, when we speak with a customer service agent, when we want to get a free gift card, and even when we go to the hospital.

We’re all pros at taking surveys… and we all know when we’re taking one that’s TERRIBLY designed.

For me, if I don’t feel like I can answer the questions, or if it gets too long or overly annoying, I’m out.

And that organization just lost a respondent.

I don’t want that to happen to you — because in education, surveying our stakeholders is SO important. It shows that we value our stakeholders’ opinions, feedback, and experiences.

We can’t afford to lose respondents because of iffy survey design.

Here are a few of my tips for upping your survey game:

1. Ask only what’s really important.

Make a list of what your team is wondering about or what the impact of your proposed projects/plans might be before you draft your survey questions.

Keep it short and sweet … if it’s not related to those things, don’t include it.

2. Reach respondents where they are.

Think of all of your touch points with your key stakeholders. Students may be log in for online class, families may check social media for updates, and all of your stakeholders may access meal sites.

At all of these venues, you can easily ask about needs, satisfaction with the school’s efforts, or other questions you may have.

You can also get feedback through polls in Google Classroom, Zoom, via text message, or even on social media.

3. This may seem obvious, but … make it easy for respondents to actually answer your questions.

Keep the language clear and simple so a person of any reading level can understand it.

Never ask about more than one topic in a single question, and try to avoid giving a neutral middle answer option when you can.

(In both of these cases, it’s very hard for you to actually learn anything from the data.)

And of course, if you work with communities who speak languages other than English, find a way to translate your survey into their language.

Translation is a much tougher process than it should be, but it is essential for making all of your families feel valued and for hearing from your entire community, not just one subset.

All that being said, I feel the same way about you – my colleagues, clients, and readers.

I want to know what’s important to you and what would be helpful for me to cover on the blog.

I hope that you’ve been inspired by this post and will take my brief survey below.

I appreciate your feedback and will use it to generate future content for you!

Loading…

cplysy

Watch Our Conversation

My Introduction to Dusty Shelf Reports

Reasons Evaluators Should Be Learning About Data Visualization and Reporting

Resources Someone Just Starting Out in Evaluation (and Others) Can Use to Get Proficient in Data Visualization

Creating a Dataviz Wall of Fame

Techniques for Engaging Project Stakeholders in the Reporting Process

Easy Fixes for Reports

Going Beyond the Report

Structure Your Report

Design Each Page

How to Obtain Feedback on Your Work

Masterminds

Collect Feedback After Trainings and Courses

Inspiration for Developing Blog Posts

Book Recommendations

How to Structure Your Day/Week to Enhance Productivity

My Husband is a Stay-At-Home Dad

We Bought a House with a Quiet Office

Be Smart About Scheduling

Creator vs. Manager/Delegator

Morning, Afternoon or Late-Night Person

Learn More

Your Turn

Setting the Stage

Activity

Why?

Footer

Follow our Work