allblogs

Sep 30 2020

Independent Consulting in the COVID Era

So I’m launching a new course today: Independent Consulting Jumpstart.

It’s a blended course, the first part is a collection of self-paced activities split into 4 modules (business goals, market finding, pricing, and marketing). The second part is a live group coaching sprint (a series of 4 calls), the first call of the series will take place over Zoom on October 6.

I want people who could stand to benefit the most to join, so it’s $99 for everything and includes forever access.

When I first started consulting I had the opportunity to work out of a busy co-working space. And while there I was surrounded by other small business owners. Some were peers with just as little experience and others were mentors who had lots more.

During this time I would also attend a good number of networking events and meetups. Pretty normal behavior for a new consultant. These spaces and events provided something I wouldn’t get working out of my home…people.

Now as life continues to move forward in the COVID era, the network events, meetups, and co-working spaces that fueled my personal small business growth are either nonexistent or completely restructured.

Independent consulting was already on the rise. The mix of enabling technologies, workplace culture changes, and shifting management philosophies were turning full time employees into self-employed consultants at a fast rate.

And that was before COVID pushed the unemployment rate sky high, which can only lead to accelerating that pace.

Civilian Unemployment Rate in the US (accessed September 28, 2020)

Many new consultants (self-included when I was at that stage) tiptoe their way into business. And if you have plenty of time for self-discovery, that’s a fine way to do it. But if you want to design a business that makes enough money to keep your family fed, you should have a better strategy.

Over the last few months I’ve been having more calls with prospective consultants. Just offering advice and a little moral support from a distance. And for the people I’ve connected with thus far, the advice seems to be helping.

This course is an attempt to scale that support. To offer insight I have collected over the years and create temporary spaces for peer connection/support.

If you are currently in the position of starting your own independent consultancy (or thinking about it) I encourage you to check it out. It’s not a long course, but it asks hard questions that might take some time to answer. If you can’t make the coaching sessions, they will be recorded, and I plan to lead another series sometime in the next couple of months (which you would also have access).

And if you have a friend trying to start an independent consultancy, please forward this along.

Cartoon Gallery on Pinterest

Trying again to make it easier for everyone to keep track of my cartoons. I’ve been drawing more and they are ending up in different spaces. So I’ve decided to pick back up on using Pinterest.

Each time I create a post, either here or elsewhere, that includes new cartoons I will simultaneously save the cartoons to Pinterest. That way it will be easier to scroll through without all these pesky words getting in the way.

This one is apparently the Pinterest favorite at the moment, wonder why?

“Book Tour”

Another perk from putting about a book of cartoons, I have been getting invitations to be on internet shows and podcasts.

I’ll be joining Pieta Blakely, Cynthia W. Rojas, and Rebecca Tuttle on their Facebook Live show Coffee Time with Masterminds this Friday (10/2) at 10:30 AM.

Sep 29 2020

7 Questions to Evaluate Design Thinking

Design thinking is much more than sticky notes, whiteboards and creative exploration. It’s impact can be felt in the outputs and outcomes tied to actual product or service and much further if we allow ourselves to focus on that.

Here are 7 questions that we ask of design thinking that focus on the learning outcomes and uncover the true impact of creation, design, and execution, which is a big part of what design thinking is all about.

By asking these we can better tap into the true return on investment of design thinking as a transformative approach to learning, not just product or service design.

What do people learn in the process of engaging in design thinking?
What new skills to people acquire, develop, or refine through design thinking?
How are the lessons from engaging in design thinking applied to other subsequent products?
What is the effect of design thinking on the mindset of those involved in a design-oriented project?
How does the co-design process influence team development, cohesion, creativity, and innovation performance?
What role does design thinking play in shaping the innovation culture (e.g., creation, execution, delivery, and evaluation) with an organization?
How does design thinking contribute to the implementation of innovations?

Evaluating the impact of your products or services is always important, but if you focus only on that you will miss some of the biggest benefits that design thinking offers your organization when done well.

If you need or want help in learning how your team learns and amplifying the effects of design thinking, contact us and we’ll help you out.

Note: This article was inspired by a recent post on our sister blog, Censemaking, which focuses on ideas, commentary and issues tied to innovation.

Sep 28 2020

Why I love Perusall

This semester, I started using Perusall for all my assigned
readings in my evaluation course. In this course, students are often learning
for the first time what evaluation is while at the same time working with a
local organization to design an evaluation for them. There is a lot of ground
to cover while simultaneously giving them enough time to work with their
clients.

What is Perusall?

Perusall is a collaborative
reading application. Students and instructors read articles or websites or
watch videos on the app and can comment chunks of text or time stamps in the
video. These comments—or annotations, as they’re called on Perusall—can have
simple text formatting, code snippets, emojis, images, links, and videos. Then other
students or instructors can reply to that comment to make it a thread or
conversation. Comments can be upvoted, and annotations that are questions are
automatically recognized and other students can indicate they’d like the answer
to this question too. In that way, the scoring is a bit “gamified” to encourage
high quality annotations that others find useful or insightful.

Helpfully, the Perusall platform is free for students,
instructors, and educational institutions. Although I do not use a textbook for
this course, students can purchase their textbooks through Perusall. This would
be problematic at our institution since we do textbooks via rentals to students
to balance textbook costs across students. However, I do not plan on using this
platform (yet) with my undergraduate students and I currently do not use
textbooks with my graduate students.

A view of Perusall. The article (Wanzer, 2020) is on the left, with highlights indicating conversations. The purple highlight is the current conversation on the right, and the yellow is the next conversation that a student started. On the right, I started a conversation and four separate students’ responses are shown. Someone upvoted the question and someone upvoted the first reply. On the far left of the article, there’s a picture bubble with my avatar indicating I’m the only one actively reading the document currently.

Why use Perusall?

After seeing a demonstration of Perusall by Eric Mazur, I
was hooked. This made reading fun, engaging, and not so lonely of a process. During
my first year of graduate school I did not quite know how to read effectively,
and I imagine most of my students feel much the same way. Perusall helps students learn to read complicated
texts together. They ask questions, they make comments, and I can monitor and
contribute to the conversation to keep everyone on track.

As a result, this year’s cohort seems to have grasped what
evaluation is far quicker than last year’s cohort. To be fair, there could be many
other possible explanations; there are some differences in the make-up of the
cohorts or perhaps my teaching has improved from last year to this year.
However, personally I think Perusall has made a world of a difference. Here are
some benefits that I have noticed:

I can save time in class by focusing on
activities applying what they learned in the readings rather than
discussing the readings themselves. We sometimes go over the readings in class,
but this has freed me up to focus on other things which has also helped them
grasp evaluation much more quickly. However, note that this will take them more
time out of class and to plan the workload accordingly.
Students can read at any time but still
benefit from reading “with” peers and me. I encourage students to check
into Perusall once a day to catch up on the latest conversation. They also get
notified when someone tags them or responds to their conversation. As a result,
this does not become a task that students just read and complete. It also leads
to more conversation than the typical LMS discussion board provides. If this
were an entirely asynchronous course, this would be highly beneficial to
promote student-to-student and student-to-instructor interaction.
The conversations we are having are deeper
than most conversations I ever experienced in class-wide or small-group discussions
during graduate school. Even if the entire class period gave time to have a
discussion, it was never enough to really get to everything. This lets us
review and discuss every piece of the reading: not only the big major take-aways
but also some of the nitty gritty or fun tidbits.
Although I can guide the conversation,
students can also take leadership of the conversation. Sometimes I will start
conversations to get students thinking about the major take-aways of the
articles. However, I’ve done this less and less as the semester has progressed
because I’ve found they are grasping the take-aways themselves better over
time. They can also introduce new and exciting conversations that I had not
even thought of myself!

What are the potential downsides?

I think there are a few, and some that are specific to my particular
situation, but overall I think the benefits above outweigh the costs.

First, this is going to take students more time to complete.
Average reading time per article ranges from 30-90 minutes per student, and I assign
multiple articles a week. I had to adjust my out-of-class activities
accordingly. However, I also teach another course these students take and, except
for the first few weeks, there is far less reading in that other course.
However, I think the benefits of understanding the reading more deeply outweigh
the costs here.

Second, it adds another thing students must learn. Currently
students have to learn our LMS, Microsoft teams (for online class meetings),
and now also Perusall. That can be an added challenge. However, I think the
benefits of a platform that provides a better reading experience outweigh the
costs here.

Third, my institution will not enable the integration into
our LMS. My hope is if more faculty begin adopting this program, they will
investigate an integration in the future, but for now this becomes a separate
website that students have to check regularly. Part of the issue is that
Perusall will not meet individually with institutions to ensure accessibility
regulations are being met; instead, they provide a website with detailed information
because they are faculty-run and supported and are a free platform.

How can I get started?

I recommend watching a previously recorded webinar or signing
up for an upcoming webinar here. If
you’d like to read research about using Perusall, check out their list of
research here. Otherwise, create an
account, create a course, add your readings and students, and get started!
There is plenty of support
content to help you get started and manage Perusall.

Sep 23 2020

Unwritten Outcomes

Evaluation as a profession is built around the rational. So what about the things that seemingly defy logic?

So this is sort of a cartoon illustrated philosophical brain dump week. I’ve been diving deep into my public library’s audio book collection and thinking a lot about rationality. If you don’t want to think, just skim through the cartoons. You’ve been warned.

I remember thinking a bit about evaluation when perusing some of Daniel Kahneman’s work on behavioral economics. Economics traditionally was based on this idea that people make rational decisions. Behavioral economics calls that into question and it led to a Nobel prize for the psychologist, Kahneman.

And Kahneman’s work can now be seen transcending fields. I mean it came up most recently when I was reading a book by hostage negotiator Chris Voss.

Everything we’ve previously been taught about negotiation is wrong: you are not rational; there is no such thing as ‘fair’; compromise is the worst thing you can do; the real art of negotiation lies in mastering the intricacies of No, not Yes.

Never Split the Difference by Chris Voss

I had been waiting for the idea to creep into the evaluation world. Like for someone to write a book titled behavioral evaluation. To start making claims that people are not rational so how do we expect to create accurate rational models of the activities of people?

His central message could not be more important, namely, that human reason left to its own devices is apt to engage in a number of fallacies and systematic errors, so if we want to make better decisions in our personal lives and as a society, we ought to be aware of these biases and seek workarounds.

Daniel Kahneman changed the way we think about thinking. But what do other thinkers think of him?

But lately I’ve been feeling like saying we are not rational is just a cop-out.

Of course we’re rational. We make rational decisions all the time. But my rational isn’t necessarily your rational.

My desired outcomes might be different, and who is to say my desired outcomes will even stay fixed. Have you ever lost a game on purpose? Taking it easy on someone to help build their confidence. Or maybe another player does something that annoys you. Does changing your desired outcome from “winning” to “making sure that one jerk loses” mean you are not rational?

Maybe, just maybe, it’s not a lack of logic we’re noticing. It’s just people who don’t do what we think they are going to do.

It’s not the work of white Nobel prize winners that we need to see further integrated into our field. Perhaps though, it should be the work of feminist thinkers like Audre Lorde.

Rationality is not unnecessary. It serves the chaos of knowledge. It serves feeling. It serves to get from this place to that place. But if you don’t honor those places, then the road is meaningless. Too often, that’s what happens with the worship of rationality and that circular, academic, analytic thinking. But ultimately, I don’t see feel/think as a dichotomy. I see them as a choice of ways and combinations.

Sister Outsider: Essays and Speeches by Audre Lorde

But of course it has already.

I find myself “discovering” works well known to many. Works that have already inspired countless, and will only continue adding to those numbers over time.

I soon recognized that there was a severe disconnect between the grantmaker’s success measures and the mission and/or approach to the work of many of the grantees. I wrestled with how to provide enough insight toward demonstrating some sense of accountability for the grantmakers while also creating space to use the evaluation findings to support grantee effectiveness (e.g., delivering technical assistance).

I discovered along the way that the root of this disconnect was not a grantee’s ability to be “successful”, but the varied definitions used for success. While struggling to figure out how to effectively address this observation, I recalled the words of Audre Lorde, an author whose writings I fondly admire and reflect on regularly. In other words, many grantees are trying to “crunch” their work into someone else’s definition of success, and in that process, their true efforts and impact go unnoticed.

The Day Audre Lorde Inspired Me to Reconsider the Definition of Success by Alison T. McMcNeil

So wait, where was I?

Oh yes, rationality.

I think this is the lesson that’s now stuck to my heart.

Desired outcomes are as infinite as our motivations. In a single person they can change quickly with new context, or gradually over time. In groups of people they are never fixed.

As evaluators, we need to seek out these motivations and desired outcomes, both written and unwritten. Because rational exists, and that rationality is essential to understanding how programs work or do not work.

It just might not be the rational represented by the official program theory of change.

We believe the one who has power. He is the one who gets to write the story. So when you study history you must ask yourself, Whose story am I missing? Whose voice was suppressed so that this voice could come forth? Once you have figured that out, you must find that story too. From there you get a clearer, yet still imperfect, picture.

Homegoing by Yaa Gyasi

Sep 22 2020

Ending Dusty Shelf Reports: Interview on James Pann’s YouTube Show

I recently had the opportunity to be a guest on James Pann’s YouTube show to talk about reports. James is an Associate Professor, program evaluator, and psychologist who specializes in evaluating health, human service, and educational programs.

Watch Our Conversation

Here’s a recap of our conversation:

My Introduction to Dusty Shelf Reports

I recently had my 7^th year anniversary of working for myself. I started out as an evaluator and I always loved the data viz side of my job. Some people love doing logic models or lit reviews, me I love poring through spreadsheets trying to figure out the pattern.

Early on in my career, I was throwing all my effort into a report, staying late, working on the weekends, doing whatever it took to do my best work. My boss sat me down and kindly said, “Ann, you’re a great team member and I’m so glad to have you. I’ve noticed you’re working really late on this evaluation report. It matters, sure, but I don’t want you to sacrifice your personal life for a dusty shelf report.”

I’d never heard that term before and I instantly felt crushed, but also thinking, “there’s got to a better way- this can’t be the standard”.

Reasons Evaluators Should Be Learning About Data Visualization and Reporting

If you’re working in academia or are a student in grad school students, then you absolutely need to write academic reports. But if you go outside of academia, there are many other workplace settings that have their own standards. The bar is being raised so much (in a good way) in thinking beyond the report. Can we also have interactive dashboards? Infographics? Amazing things are being done.

The tricky part is that none of us were trained for how you go from really technical reports in academic settings to non-technical reports, non-technical audiences and really quick turnaround times.

Resources Someone Just Starting Out in Evaluation (and Others) Can Use to Get Proficient in Data Visualization

For a student just getting started or if you’re new to the evaluation field, you can take one of my courses (hint, hint, nudge, nudge).

Soar Beyond the Dusty Shelf Report is our free mini-course where you’ll gain practical techniques that you can apply immediately to improve your visuals.

For reporting the best place to start would be to do a self-assessment. Ask yourself: what is the impact of your reports (if any).

There are four red flags to really look for with reports (or slideshows or dashboards, etc.). Look out for these:

No Response
- No response at all after you send it out.
Promise to Follow-Up Later
- You get a response that says, “Thanks, I’ll let you know if I have any questions.”
“Compliments”
- You get a response that says, “Thanks, we can tell that a really technical team worked on this report. It’s very detailed and thorough.”
Won’t Read It
- They ask for another format entirely like a slideshow, one-pager, etc.

Creating a Dataviz Wall of Fame

One thing I did for myself early one that was so helpful was to create a data viz wall of fame. I used to look for examples of great graphs, dashboards, infographics, etc. and would print them out and plaster them on the wall above my desk.

Nowadays, you could have a Pinterest board, folder on your computer or favorite them on Twitter. Start by actively collecting examples that you love but then push yourself to the next level by thinking, “What’s the rubric for this? Why is it good- is it the writing, the type, etc.”. Once you can identify why it’s great, you can start to replicate it.

There’s a lot of great podcasts out there too. The ones that come to mind are:

Data Viz Today with Alli Torban

Data Stories with Enrico Bertini and Moritz Stefaner

Data + Love with Zack Bowders

PolicyViz with Jon Schwabish

Techniques for Engaging Project Stakeholders in the Reporting Process

I co-authored an article on using Data Placemats, which is a technique you can use to engage project stakeholders in the reporting process.

I want you to step away from the idea of waiting until the end of the project to give the completed report/slideshow and hope that it gets used. Instead there’s a three-step process you should follow:

Develop data placemats with preliminary findings- traditional format.
- Make a data placemat. Use Word or PowerPoint and lay out one graph per finding.
Meet with stakeholders and have them explain their findings in their own words; you’re a facilitator, not a lecturer.
- Hold a meeting with the stakeholders where you pose discussion questions and then sit back and listen. Walk them through how to walk through the data and have some of the ‘lightbulb’ moments themselves.
Write the final report- storytelling graphs- using their interpretations.
- Take all that data you collected from them and their insights and put that into the report.

Easy Fixes for Reports

Here are the most common mistakes–and easy fixes–that I see.

Going Beyond the Report

You only have a report. A report can’t meet every audiences’ need. You need one product per audience.

Structure Your Report

Not starting with the “So What?” The most important information is buried in the report. Put the “So What’ on the first page and get your audience in the game right away. Don’t risk losing your audience.

Not following brand guidelines. Use your fonts and colors, especially think about using your recipients brand guidelines. It’ll take you about 10 minutes but shows that you’re willing to go the extra mile and it looks more professional.

Design Each Page

Not aiming for 1+ visual per page. I define visuals very broadly. You might use the logos of organizations, or include a timeline to show the numerous steps taken.

Forgetting to lower the reading level. The average American reading level is 6th through 8^th grade. Plug your text into a readability level checker and make sure your reading level matches your audience. Aim to be about two levels below your audience’s educational level. For example, if most of your audience members have completed Master’s degrees, then aim for two levels below that (high school).

How to Obtain Feedback on Your Work

When you blog and YouTube like I do where your work is front facing, people will tell you! I remember posting my first YouTube video and the first comment was from an anonymous person who wrote in all caps, saying they couldn’t hear anything over the clicking of my keyboard and said it was such terrible quality. I didn’t even know people used a microphone! I watched YouTube videos on mic and lighting comparisons to get to my setup now.

I did a webinar a week or two ago and my phone number is on my website (I forward my business phone number to my cell phone). This guy called and said, “I paid for your webinar you did yesterday and I have one problem with it”. I instantly went into flight or fight mode wondering what he was going to say. He went on to say how most presenters just read off a script but you didn’t, it was one of the best he’d seen. People will just call and will tell you very openly how they feel about your work so you have to have a poker face.

Masterminds

I’m also in a few masterminds with some other business owners where we act as each other’s sounding boards and brain trusts. I’ll throw out ideas to them and they’ll flat out tell me that it’s not a great idea. It’s really helpful to have a peer group that you trust.

Collect Feedback After Trainings and Courses

When I do trainings, I collect feedback forms or do an internet survey. In my larger projects, I build it in to have a debrief. They turn into problem solving for when hiccups may happen in the future. In shorter projects, I look for repeat clients or referrals. If I never hear from them again, that would worry me.

At the end of my courses, student receive a course survey. I also have behind the scene data for course students that let me know how much of a course they’ve completed, how often they’re logging in, if they’re watching all of a video or not and more. I can see which videos people are re-watching or even which sections they’re re-watching which tells me where they’re getting stuck.

Inspiration for Developing Blog Posts

I usually write blog posts on questions that workshop or course participants are asking. If everybody is struggling on how do I make sure my colors are accessible, that’s a great blog post. I also do a lot of before/after makeovers which help people see all the little steps that can make a big difference. I try to show the behind the scenes thought process so that people can then replicate it in their work.

More recently I’ve been doing interviews to try and get to know the people in data viz better and then I turn that interview into a blog post.

Finding a topic that you personally care about is key. I don’t think blogging is supposed to feel like work. You could monetize your blog but it would take a lot to make a full time living from a blog, but it’s mostly a hobby and supposed to be fun. I love the idea of thinking about what’s interesting to you and what do you want to explore more.

Book Recommendations

I have gifted all of my siblings-in-laws, Your Money or Your Life by Vicki Robin. She published it back in the 90s for the first time and just did a re-do a couple of years ago. It’s a financial book kind of, but it’s more about thinking about your life energy. You’re only alive for so long and you have a limited number of days, how can you make the best use of your time? It’s about trying to avoid just consuming things that give you temporary happiness, but thinking about how do you create for long-term, life satisfaction?

My husband I listened to the audio version during a long car drive and kept pausing it to talk about the insights we were having. She says to not throw away your life in a job that’s only decent, you need to aim for a great job and set up. Or maybe you should stay home and be a parent. This is what inspired us to have my husband quit his job ad have our family join me on the road while I traveled teaching. This book clearly changed our life trajectory.

Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones by James Clear is so actionable for everybody in every industry. I think everyone can learn something from it.

James: Steve Kotler’s stuff on flow like the Rise of Superman: Decoding the Science of Ultimate Human Performance. Also, Bernardo Castro has a philosophy book on idealism vs materialism. It’s not some academic nonsense, but it’s very practical. I think he’s one of the most important writers right now.

How to Structure Your Day/Week to Enhance Productivity

That’s a tricky one because I think there’s perception that Ann Emery must wake up at 5 am, run 10 miles, then make a green smoothie and work, work, work until midnight. She never sleeps and must be miserable, clearly. In reality, I get a lot done but I also rest and have fun. I don’t see those as opposing dichotomous topics at all.

I think you can have this wonderfully fulfilling personal life and achieve a lot professionally at the same time. I’m into minimalism in terms of thinking about what really matters in your work and personal life (which ideally are the same thing!). But really thinking about what matters and let go of everything else.

My Husband is a Stay-At-Home Dad

My husband let go of his very good salaried job with a pension and top-secret security clearance. We let that go so we no longer have to do things like both waking up with alarm clocks, rushing the kids off to daycare, both working and then figuring out who makes dinner, etc. We now don’t set alarm clocks and get enough rest each day.

We Bought a House with a Quiet Office

Now if I have to set an alarm, I think to myself, “My life is out of control! Why did I schedule something before 10 am?” We purposefully stopped fulltime travel due to COVID and so when house hunting in Florida, we were looking for a very specific home. We wanted a one level house with a bonus space I could use as an office. It’s my little retreat- I go to my office, get all my work done and then I go downstairs and go ‘home’.

Be Smart About Scheduling

I used Calendly for scheduling meetings and I’m only available from Mondays at noon until Thursday. I don’t book anything on Monday morning, that’s a terrible way to start the week! And I don’t boo anything on Fridays, because if my brain is tired, I’m not going to be any good on a Zoom call. Fridays are usually a day for myself to write a blog post or record videos for online courses. It feels really good to end the week like that, doing something creative for yourself.

Creator vs. Manager/Delegator

Identifying whether you’re a creator or more of a manager/delegator and making sure you’re not trying to go back and forth. Instead of trying to spend one hour writing a blog post (creating), then a 30-minute meeting about a project and then spend one more hour writing that blog post. I try to really separate it and have a half day of uninterrupted to do the actual, creative work. Then I try to book all my meetings, back to back at another time.

Morning, Afternoon or Late-Night Person

I’ve listened to whole podcast episodes about being careful to know if you’re a morning, afternoon or late-night person. I used to be a late-night person and would do my best work at 9 pm-midnight. But now I’m a morning person so making sure that my mornings are my uninterrupted time. I rarely look at emails and instead am building courses and writing. In the afternoon, I take calls. You really need 100% of your brain energy for the creative work.

James: I think it’s great to get good sleep, that’s probably the first and most important thing. Getting that honed in has been huge.

And then being okay with taking breaks during the course of the day and actually building it in. Making sure that you’re going through a cycle and not just powering through.

Learn More

Subscribe to James Pann’s YouTube show.

Your Turn

Comment below: Which reporting tips were new to you? Which tips are you going to put into practice? Have reporting tips of your own to share?