cplysy

Dec 19 2023

New Infographic: Types of Interview Guides

This article is rated as:

Eval Academy just released a new Infographic: “Types of Interview Guides”

Who’s it for?

This infographic is for anyone looking to learn more about the different types of interviewing or for those who are unsure which type of interview guide to use.

What’s the purpose?

This “Types of Interview Guides” infographic will help you to:

Learn about the key characteristics, advantages and potential use of the different types of interview guides.
Choose which type of interview guide to use in your qualitative data collection.

What’s included?

A one-page, downloadable infographic as a png file.

Download the “Types of Interview Guides” Infographic now!

Learn more: related articles and links:

You can learn more about collecting data with professional and ethical conduct in the following Eval Academy articles:

Other Eval Academy resources that you might be interested in checking out:

What do you think of our new template? Let us know in the comments below!

Dec 19 2023

How to Link Surveys in Qualtrics with a Participant ID

This article is rated as:

This is a step-by-step guide for linking your surveys in Qualtrics with a participant ID to assess change over time. There are ways to set up these surveys to automate the process to a greater degree than what will be covered in this article. Qualtrics has a support article that has more details for how to do those things such as automating Qualtrics to randomly generate participant IDs or setting up a way to automate the survey distribution to an email list.

The steps below outline ways to for link your surveys with a participant ID that is manually entered by participants, rather than automatically generated, and assigned. Please note that you can only link surveys using a paid version of Qualtrics.

What does it mean to link your surveys?

Linking surveys together means that you’re connecting the data from all your surveys in a way that lets you track an individual respondent. For example, if Kay fills out survey 1 about how they like dogs more than cats, and then changes their mind 6 months later when they take survey 2, you’ll be able to see that pattern in the data on an individual level if the surveys are linked.

Why would I want to link my surveys?

If you want to know how individuals are influenced and change their behaviours or views over time, linking surveys can help you see that in the data at a greater detail than if your data was only in an aggregated form. Linking surveys is a useful method for collecting longitudinal data.

For example, let’s say you were trying to measure the influence of an ad campaign that is trying to get people to like cats more than dogs. You can do a regular survey BEFORE the ad campaign runs and find the total percentage of people who like cats more than dogs. And then you can do another regular survey AFTER the ad campaign to see if that percentage has increased. However, your results won’t tell you if most of the people who changed their opinions were between the ages of 30-40. It also wouldn’t tell you if some people who liked cats more than dogs changed their minds too. Linking surveys can allow you to see more details in the pattern of your results.

Does linking surveys reveal a participant’s identity?

Linking surveys lets you track individual data confidentially. You can set up linked surveys so that any personally identifying information is not linked directly to an individual’s response. This means that when you are looking at the survey data, you’ll see the answers given by participant 1, but you won’t know their specific name or contact information. A confidential survey is different from an anonymous one. Linked surveys cannot be anonymous because they generate a list of everyone who has taken the survey. If you are using this method, it is important to help your participants understand that while their responses are confidential, they are not anonymous.

To learn more about the difference between confidentiality and anonymity in evaluation, check out, “Your information will be kept confidential: Confidentiality and Anonymity in Evaluation.”

Before we get to the details, I want to remind you that it is best practice to test the survey functionality and pilot it prior to sending it to participants.

How to manually link surveys in Qualtrics

Setting Up Survey #1

1. Create your first survey in Qualtrics as if you are setting up a regular survey.

2. Create a survey question that will determine the participant ID. This is the ID that a participant will need to remember and enter to access the second survey.

For this guide, we will use the participant’s email address as the ID. We chose to create a text entry question with an email address validation that asks, “What is your email address?”. You can choose a different identifier like a phone number or a name – something that the participant will remember easily. Make sure you make this question mandatory. It is recommended that you put this question near the beginning of your survey so that participants will fill it out even if they exit the survey early.

3. Go to the “Survey flow” tab.

4. Select “Add a New Element Here.”

5. Select “Embedded Data.”

6. Type in a label for your participant ID. Since we’re using email addresses, we just wrote, “Email Address.” Do NOT select an option from the drop-down list.

7. Next, click on “Set a Value Now.”

8. For “Set a Value Now,” you will need to choose the survey question that you had created for your participant ID. In the drop-down list, select “Insert Piped Text,” then “Survey Question.” Find the question you created for the participant ID. There will be one more panel that appears with options. Select the box that has the actual question. For us, it is “What is your email address?” Do NOT select “Question Text.”

9. Select “Apply.”

10. Next, go to the “Workflows” section.

11. Select “Create a workflow” and choose the option “Started when an event is received.” Another page will pop up that will list different types of extensions.

12. Choose the extension “Survey response” by Qualtrics. This means that you are now creating a workflow within Qualtrics that will be triggered when someone responds to your survey.

13. In the next set of options that appear, make sure that “Newly created responses” is checked. You can make selections under “Additional response types” as you prefer. Select “Finish” when you are done.

14. Next, select the plus sign (+) and then select “Add a task.” Another panel with an extension will appear.

15. Select the extension “XM Directory” by Qualtrics.

16. Select “Add to XM directory.” A new panel with fields will appear. What you will be doing at this point is creating a contact list for participants that will be stored on Qualtrics. This contact list is how Qualtrics will recognise the participant IDs when they are entered to access the second survey.

17. Select the drop-down box for “Contact List.” From there, you will select “My Library,” and then “New Contact List.”

18. Create a name for your participant contact list. Then select “Save” to exit the window.

19. Next, select the drop-down box for “External reference ID.” From the list, select “Embedded Data Field” and then select the name of the label you created for your participant ID in step 6. For us, the name we chose was “Email Address.”

20. Leave the other settings under “Contact update options” as the default settings. This means that “Save or update it as embedded data to your XM directory contacts” is selected and “Update recipient from survey response” is checked. Save to exit the window.

21. Your first survey is now ready to be distributed. After publishing your survey, make sure you use the “Get a single reusable link” option from the “Distributions” tab. That will give you a single anonymous link that can be sent to all the participants. This survey shows as a regular survey. When the respondent submits their response, their information will get added to the contact list associated with your survey.

Congratulations! You have just set up your first survey! Try testing it out and if you encounter any problems, contact Qualtrics Support and someone will help you troubleshoot your issues.

In my experience, it takes Qualtrics a few minutes to update your survey with the changes you’ve made. If you don’t see your changes right away, wait and try again before contacting Qualtrics Support.

Setting Up Survey #2

22. Create your second survey like you would for a regular survey. This one doesn’t need a specific question for the participant ID. If you are asking the same questions as survey #1, I would recommend copying that survey and editing it to make sure your questions are worded in the same way.

23. Go to the “Survey flow” tab and select “Add a New Element Here.”

24. Select “Authenticator.”

25. Select “Move” on any question blocks you have and move them underneath the Authenticator element. If you have multiple question blocks, be sure to move and arrange them in the correct order for your survey.

26. “Authentication Type,” should be set to “Contact.” And under “Authenticate Using Contact,” select the same library you used in step 17 of survey #1. When you select the library, another drop-down list will appear for contact lists. Select the contact list you created in in step 17 of survey #1.

27. After selecting your contact list, another section called “Authentication Fields” will appear. Under this section, select “External Data Reference” in the drop-down box, then type in the label you chose for the participant ID you selected in step 6 of survey #1. For us, we had chosen “Email Address” so that is what we wrote in this box. Leave the “Password” and “Pre-fill” boxes UNCHECKED. Apply your changes.

28. After publishing your survey, it is now ready to be distributed. When you are sending it out, make sure you use the “Get a single reusable link” option from the “Distributions” tab. That will give you a single anonymous link that can be sent to all the participants. After receiving the link, they will be taken to a page where they will have to enter their participant ID before accessing survey #2. Requiring participants to enter their participant ID ensures that their survey #2 responses are linked to their survey #1 results.

Anyone who has not taken survey #1 will NOT be able to access survey #2, unless you add their contact info manually to the corresponding contact list in Qualtrics. This is the contact list you created in step 17 of survey #1.

And that’s it! You have just linked your second survey to your first! If you don’t get the result you want after testing it out, contact Qualtrics Support and someone will help you troubleshoot your issues. Sometimes it takes a few minutes for Qualtrics to update your changes. So, if you don’t see your changes right away, wait a few minutes to test again before contacting Qualtrics Support.

Dec 18 2023

Top 23 Dataviz Resources of 2023

9,200 total participants in my dataviz courses so far (and ~25,000 taught in-person and virtually over the past decade).

A dozen in-person workshops (with trips to Dubai & Tanzania) and several dozen virtual workshops.

32 new blog posts: 28 from me, and 4 from guest authors.

2 more YouTube videos (115 total).

2 more podcast interviews.

2 more babies.

6 months off.

What a year.

Top 23 Dataviz Resources of 2023

Want to do some year-end learning as 2023 winds down?

Here are my favorite data visualization resources from the past year.

The Progression of Sue Griffey’s Year-End Infographic: Sue Griffey explained how she edited her infographic and we discussed it together during Office Hours. I love going behind the scenes like this!
Watch Out for Mars! 6 Data Cleaning Steps to Save You Millions: Hudson Kelley teaches us how to check for duplicates; check for changes in the survey instrument; check for outliers; use counts; recode variables with IF statements; and combine datasets with lookups.
How to Influence Others with Your Data: SuperDataScience Podcast Interview: I was invited to speak on the #1 podcast in the data field. Host Jon Krohn asked me to summarize some of my favorite tips for data storytelling, spreadsheets, reports, and presentations. This is a great listen for people who are new to my work.
Embedded Legends Aren’t Enough: I see this accessibility mistake way too often. Take a peek and make sure you aren’t messing up your graph titles and legends.
How to Visualize “Overall” Data or Averages in Bar Charts: Probably my most practical post of the year.
39+ Amazing Graphs You Can Make in Excel: Probably the community’s favorite post of the year. This isn’t an exhaustive list, and I’ll add to it someday. In all my spare time.
3 Simple Steps that Took My Graph from Good to Great: In this before-after makeover from a museum evaluation project, Maia Werner-Avidon teaches us how to use grouping, spacing, and icons.
How to Make Great Graphs in Excel: 4 Levels of Excel Vizardry: Probably my personal favorite of the year. Everyone wants to jump into advanced graphs, and then they flail and drown. In this post, you’ll see which skills you should start with.
How to Analyze Nonprofit Data with Excel’s Pivot Tables (No Formulas Required!): I shared the recording and materials from my how-to workshop at the Good Tech Fest conference.
Building a Business that Fits Your Family: Disrupt Your Money Podcast: I was invited to speak on Meg Wheeler’s financial podcast. I shared a lot of personal and business details.
From Formulaic to Meaningful: Constructing a Useful “Table of Contents” Page for an Evaluation Report: Barbara Klugman shared her before-after makeover on the discussion boards inside Report Redesign, and I invited her to write a blog post so others could learn from her, too. Thanks, Barbara!
How to Visualize Multi-Year Patterns: Another super-practical post. It would’ve been a personal favorite if people on social media didn’t prefer the terrible, horrible, before version better than the redesigned options. Sigh.
Two Types of Tables: Datasets vs. Tabulations: If everyone followed this advice, I could retire. Someday!!!
Two Types of Datasets: Contiguous vs. Non-Contiguous: The most important time-saver I’ll teach you all year.
Two Types of Tabulations: Formulas vs. Pivot Tables: To my knowledge, none of the Excel bloggers worldwide have written about the implications of formulas vs. pivot tables for data visualization. Maybe 5 people in the entire world will appreciate this advanced, niche post. Are you one of those 5 people?
How to Make Your First Tableau Dashboard: It’ll take less than an hour, promise.
How to Make a Series of Matching Dashboards in Excel: I gave away all the behind-the-scenes secrets in this one. Companies pay me $10,000 – $30,000 for these types of automations in consulting projects. Please, just follow these instructions and do it yourself. I’d love to retire someday.
How to Make Interactive Dashboards in Excel: You’ll learn how to link Excel Tables, pivot tables, pivot charts, and slicers.
Redesigning a Thesis Chapter: Farihah Malik does the very hard work of applying what she learned in Report Redesign to academia.
Use Icons to Visualize Data (Not Just Decorate): An advanced, niche post that maybe 5 people worldwide will appreciate. Maybe you’re one of them?? It had to be written.
How to Visualize Small n’s with Icon Arrays: A quick before-after table makeover. Please steal this idea.
Bring Technical Tables to Life: A real (sort of) before-after makeover from when I keynoted the National Birth Defects Prevention Network Conference in Atlanta this summer.
Don’t Start from Scratch! Make One of These Dashboards Instead: A few of my favorite dashboard case studies, all in one place, to make them easier for your to find.

Your Turn

What types of tutorials should I create in 2024?

Comment below with your requests!

Dec 13 2023

Finding Our Stories through Digital Storytelling

The act of creating our stories and sharing them with others is a powerful way to better understand our experiences. I realized this in a training on digital storytelling with the Institute for Social Science Research at the University of Massachusetts Amherst.

I’ve long used the metaphor of storytelling in my work as an applied researcher and evaluator. The way that we ask questions, conduct our analyses, and represent our results all convey a message or story. As a qualitative and equity-focused evaluator, my aim is to tell the stories that participants in the study want to tell about their own experiences — what meaning they are making, what is important to them, what they want the audience to learn. But it occurred to me while taking this course that we don’t always fully know the message we want to share about our experiences. And the act of creating the story — in written narrative, in visual art, in film or comics or otherwise — actually helps us find our story.

Take, for example, my own digital story on the cocktail of emotions I felt when my mom was diagnosed with cancer. I began writing the story, intending to share a message about the incredible support I received from my community of people. However, through the process of creating my story, I was able to come to the meaning I needed to make about my experience — about the numerous, shifting, and often contradictory emotions I felt through my mom’s diagnosis, treatment, and ultimately recovery.

https://medium.com/media/415589ed81d66316310177aa0c76efae/href

So, what is digital storytelling? How might we use it in learning and evaluation? And what are important ethical issues we evaluators need to consider?

What is digital storytelling and why use it?

Digital storytelling is a method often used in learning and evaluation. It invites participants to identify, create, and produce a 3- to 5-minute video that shares a personal story through an artful combination of narration, images, and sound. The videos produced can then be used to collectively identify and analyze themes to answer learning questions.

Digital storytelling is a form of critical narrative research, which means that participants engage with their stories as a way to critique wider cultural, political, and economic phenomena. It’s easy to see how digital storytelling can also be used as an intervention, education, and advocacy in fields as wide as health/reproductive justice, education, work/labor rights, and more!

Digital storytelling can be used in learning and evaluation many ways. Using Cocktail as an example, the creation of my digital story was a form of data generation itself. As I was writing the narrative, reading it aloud to my peers, and getting feedback from the group, we shared insights about our experiences with the illness of a loved one. When the video was complete, and I showed it to the group, it became an elicitation device — a conversation starter for further discussion on the topic. Our collective of digital stories could even be shared publicly — for example, to advocate for supports to family members and caretakers or to educate health professionals!

How can I use digital storytelling?

Digital storytelling is a flexible method that can be adapted to meet the needs of the participants in the study. For example, if participants are based around the country, the process can be facilitated online or in a hybrid format. If participants have limited digital literacy or technology access, or a more professional appearing video is needed for advocacy purposes, a collaborative process of creating the stories can be used between the researcher and the participant (and maybe even a professional photographer and video editor). And digital storytelling can be used as one of multiple methods in a mixed-methods study.

Just be sure not to simply ‘add-on’ digital storytelling! It’s a time and resource intensive method that requires care to ensure respect and ethical treatment of participants and their stories, and to produce quality stories that generate dialogue and reflection.

In digital storytelling, there are at least three steps:

Writing the story. Once a group topic or question is defined (connected to the evaluation and learning project), participants spend time drafting their storyline and narrative. This could be done within a single session or for homework. Some people may prefer to create an outline before writing — for me, I just began free-writing, and that is how I learned what my story would eventually become! The first story circle is held when participants have a first draft. Each participant takes a turn reading aloud their draft and receiving feedback from the group. Facilitators and the group can help participants strengthen their stories by identifying key messages, clarifying language, helping with tone/pacing, knowing where to cut or add content, etc.
Producing the story. When stories are finalized, participants then produce their stories. This involves recording the voiceover, creating or finding images, and selecting sound effects. We used an app called WeVideo, which was pretty easy to learn. I recorded my voiceover (it took a few tries and involved hiding in a closet to reduce background noise) and took pictures with my iPhone. I’ve even seen digital stories with hand drawn or illustrated images, which I hope to try next!
Sharing and reflecting. When all digital stories are complete, the group holds a second story circle to screen the videos. Each participant shows their story to the group. We provided feedback and reflection after each video, but it may be more timely to share 2–3 videos followed by a discussion, and then repeat until all videos are screened. Sharing the stories is the core purpose of digital storytelling, and it is what allows participants to engage critically with their ideas. In a learning and evaluation setting, facilitators could encourage participants to reflect on the conditions leading to the issues identified in the stories, what systemic issues need to change (or are changing), and what factors enable or constrain those changes. The stories could also be a powerful educational or advocacy tool for sharing with organizations, decision-makers, and the wider public.

Ethical issues and power in digital storytelling

Importance of choice. Creating my story, Cocktail, taught me one of the most important ethical considerations in visual and narrative methods — the importance of choice. I have really bad stage fright — even if it’s just reading my own story aloud to a group. Before I read my story during the story circle (which was held online), I told the group that I might turn my camera off if I start to clam up. In other situations, I’ve had to recuse myself from sharing my story personally and have found other ways to share, like asking someone else to read aloud for me. A major ethical consideration is that everyone is being vulnerable in sharing their stories — which may be quite painful. Participants need choices — to read their story aloud, to share it in writing, to ask someone to read it aloud for them, etc.

Telling a personal story that involves others. At the same time, the experience highlighted the important privacy issue of telling my own story, not someone else’s. Throughout the process of creating Cocktail, I had to continually ask myself — whose story am I telling? I made decisions about what I told in my story (and what I left out) because the cancer, the treatment, and all the emotions that went along with that wasn’t my story to tell — it was my mom’s. But I also recognized that as her primary caretaker for a time, I also had a story that needed to be told.

Confidentiality. Another essential ethical consideration in digital storytelling is the timing of obtaining permission to share stories outside of the group (e.g., as educational or advocacy materials). People may participate in the digital storytelling project with full intention to share their stories publicly. But that may change as they move through the process and realize their story is too difficult. Permission to use the stories should only be requested after the stories are finalized and screened within the group, when people have a greater sense of what it feels like to share their story and who they are comfortable with hearing it.

Fortunately for me, the course instructors were not only professionals, but were highly conscientious in ensuring we retained the power over our stories and how they were shared. Staying off camera as I screened my story, I choked up at the overwhelming support I received from my peers and the care and reciprocity they showed. Sharing it with my mom later opened up a conversation we had not had in nearly 10 years since her diagnosis. Creating Cocktail helped me to name and tell my story — of how I felt fear, sadness, and guilt, alongside love, inspiration, and community — a true cocktail of emotions.

How has digital storytelling helped you to find your story? I’d love to hear from you about how you’ve used it (or other participatory visual methods) to find and share your story, or help others do the same!

Finding Our Stories through Digital Storytelling was originally published in InnovationNetwork on Medium, where people are continuing the conversation by highlighting and responding to this story.

Dec 12 2023

More than 60 Holiday Data Comics

Are you looking for data and evaluation cartoons this holiday season? In this post I’ve pulled together all my Christmas and other holiday cartoons/comics I’ve drawn from across the years. With new cartoons added annually!

Comics from 2023

Power BI and Elves

Of course the elves don’t like PowerBI, but Santa paid a lot for that Office 365 subscription, so like or not…

Intentional Christmas Lighting

Anyone being more intentional in their Christmas tree lighting?

Baking Fidelity

Who needs logic models when you have Christmas magic?

I think it would be really hard to be an evaluator at the North Pole.

Tiny Font One-Pager

I debated using “Christmas Magic” as the way Santa would read the list. But cryogenic transmission electron microscope just felt right at the time.

Comics from 2022

Zero Carbon Santa

I wonder what Santa could use instead of coal.

Inbox Zero

Santa probably gets a lot of emails. But I doubt he’s the one reading them all. Some elf probably has that responsibility…

Ugly Sweater Contest

There was an ugly sweater contest at my office last week. I forgot about it but went to work wearing a sweater. The sweater wasn’t ugly, but it did inspire this cartoon.

Day Rate

When I talk to other indie consultants, especially new ones, my key piece of advice is that your hourly or daily rate doesn’t really matter. Budgets matter a lot more.

Take reindeer for example, they only one day a year. They might get a really good rate for that one day, but is it enough to keep them in reindeer food and shelter for the remaining 364.

Logic model is a little too simple.

Trying to think of evaluation stuff I haven’t talked about during past holidays. This one is based on the idea that a lame logic model can lead to a lame evaluation. Also, these are elves, but I think we’re a little too zoomed in for that to be noticeable.

Annual Reports

Do you think Santa makes the elves deliver a year end annual report? Seems like it would be a little bit of stressful deadline to me.

Comics from 2021

2020 2021 2022 Resolutions

I remember when the year started. Sure 2020 was a bust with COVID and all, but surely 2021 would be better…right?

So naive.

If you didn’t feel super productive or accomplished this past year, don’t worry. You are in good company with all the rest of us.

Freshspectrum cartoon by Chris Lysy.
Illustration of a Resolutions list, with 2020 and 2021 crossed out, then showing 2022.

Elfing from Home

You know if Santa allowed remote work we would see more elves around. Wonder if they are also part of the “great resignation.”

Freshspectrum cartoon by Chris Lysy.
Elf chatting with a person at the beach. "I used to work in HQ up north. But when the Big Guy started allowing remote, I hopped on the next plane."

Dissemination Magic

Every time a bell rings, a stakeholder reads your report.

Freshspectrum cartoon by Chris Lysy.
"Perhaps our dissemination strategy shouldn't rely entirely on Christmas magic."

Pot Luck Needs Assessment

I drew this cartoon even though I’m not sure if anyone is really potlucking these days. I still avoid any situation with unmasked non-family members. And everyone bringing a dish, yikes!

Freshspectrum cartoon by Chris Lysy.
"We should do a needs assessment before this year's holiday pot luck. Maybe that way we'll end up with more than just paper plates and potato chips."

RCT Christmas

It’s too easy to pick on RCTs.

Freshspectrum cartoon by Chris Lysy. "Honey, did you see this letter from the North Pole? They're going to do Christmas as an RCT. It's randoized, so we have no way of knowing whether or not the kids will get any gifts fro Santa."

Ghost Downsizing

Did you know that instead of spirits, Dickens first draft of a Christmas Carol featured three evaluators?

Freshspectrum cartoon by Chris Lysy. "Hello Mr Scrooge. My name is Ethan, I'll be your ghosts of Christmas past, present, and future. There used to be three of us but our org really cut back our evaluation budget."

Superspreader Christmas

Toyed with drawing a cartoon about Santa being high risk for COVID. You know given his weight, age, and habit of bingeing on cookies and milk.

Freshspectrum cartoon by Chris Lysy. "Mom, I'm worried Christmas might be a super spreader event. Santa literally goes into everybody's house. Think we should ask if he could leave our gifts on the porch like the Amazon guy?

It’s Christmastime in our household, so that means it’s time to switch from the unending string of professional Zoom calls to the somewhat smaller stream of extended family Facetime calls.

Zoom around the North Pole

But just because it’s the holidays, it doesn’t mean we get to take off our evaluator hats. I’ve come to believe that no matter what I call myself professionally (information designer, web designer, data analyst, President and CEO, solopreneur), I will always be an evaluator on the inside.

It’s more a way of life than a profession. A path that likely began far before you even heard of the profession.

Santa Satisfaction Survey

The trick is to not annoy too many of your family and friends with your evaluator-ness. Like perhaps when you notice a significant other not following a recipe with fidelity, just let it go.

Cookie Recipe Fidelity

But there are certainly times when you can really embrace your evaluation side. For instance, maybe your extended family drives you a little crazy. Take back your holiday with the support of a nice new theory of change.

Happy Christmas Theory of Change

Of course, this post is showing up in your email inbox a little late for this winter holiday season. Christmas is here. Kwanzaa and New Year’s will be here shortly. The Winter Solstice, Hanukkah, and Diwali have already come and gone.

Hopefully some of you were on the ball and have already administered your holiday pretests. How else are you going to be able to accurately identify any measurable increase in holiday cheer?

Holiday Pretest

Comics from 2020

Presents are Outputs

This is a cartoon I redrew in 2019 from an earlier iteration. Not sure it ever made it into a post.

Cartoon by Chris Lysy
"Before we open any gifts, remember, presents are just outputs. It's the outcomes that really matter."

Christmas Tree Chart

I believe this chart type may be a default in Excel.

Cartoon by Chris Lysy
"So I created this chart using a dual axis mix of a scatterplot and a line graph...The data is meaningless but it looks like a super cute Christmas tree, so I thought I would share."

Virtual Christmas

This cartoon is probably a good representation of my headspace last Christmas.

When PJs are Business Casual

I live in NC, we don’t have enough snow days.

Cartoon by Chris Lysy
"Every year I would look forward to our first snowy day. It was an excuse to just hang out at home in our PJs, not daring the roads to go into the office. But this year it doesn't feel quite the same."

Virtual Snow Day

I loved snow days as a kid in Maryland. And when everything was switching to virtual, the idea of snow not stopping school just sounded wrong.

Cartoon by Chris Lysy
"Dad, how many inches of snow does it take to shut down the internet? I want a snow day."

Comics from 2018

We all know why the grinch really hated Christmas.

Did you know that It’s a Wonderful Life is an evaluation story?

What is the true impact of Christmas?

Alexa at Christmas

If Charlie Brown were an Evaluator

Comics from 2017

But what are our goals here?

The meaning we get from the holiday season relies almost entirely on our expectations and the context within which we live. Maybe we just need the right goal.

Christmas is about money.

For some people and businesses, Christmas is a prime commercial opportunity. No matter how many TV Christmas specials try to counter this theme, we live in a time where decorations start going up in storefront windows around halloween.

And not because the shopping centers are just full of the holiday spirit.

Christmas is about making a political statement.

Until recently, wishing merriment around Christmas was mostly just a kind thing to do.

Happy Holidays, the thing we say when we don’t know whether the person we are talking to celebrates Christmas, is seen as a counter argument to Merry Christmas. Making every greeting a political statement.

Christmas is about spending time with family.

For many of us, the holidays are about spending time together with your small family, big family, extended family, or family of friends. Depending on who you are, and the family you celebrate with, this could be a hugely positive thing or one filled with obligation and dread. And maybe it’s both.

Christmas is about giving.

Whether it’s gifts, money, or time, Christmas is a season that puts people in a giving mood. Yes, if you so choose, Christmas can totally be about presents.

Comics from 2016

A little stats humor.

A randomista Christmas?

Holiday retrofitting.

The right gift for that person who has everything.

Keep Christmas Funded!

Yes, I am creeped out by the Elf on the Shelf, why do you ask?

Children with Her

Black Friday Inflation

Target Audience = Santa

It’s the outcomes that matter.

Some clients from Christmas

Dear Santa, can you send me a few clients for Christmas

Comics from 2014

This kind of cartoon post happens when you’re writing about data visualization in front of a Christmas tree.

An Infographic to Santa

Santa’s naughty list dashboard

Santa's interactive dashboard cartoon by Chris Lysy

Dataviz Sweater Party

At the dataviz expert ugly sweater party cartoon by Chris Lysy

Comics from 2013

Comics from the Early Years (Dates Unknown)

Rudolph and Herbie just a couple of misfits

This one is from my Redbubble shop, I created it years ago

Bonus: An Evaluation Christmas Carol (2016)

Evan Scott’s day started just like most other days.

At his desk, staring at a clogged email inbox and a stack of papers filled with things that somehow or another needed his attention.

If he had known just how much paperwork was involved he might have ignored his desires to devote a career towards helping others.

Suddenly there was a knock at the door.

“Evan, you busy?”

It was his second in command, Bo Cratchet. She had been with him when he was first pushing the idea for this grant funded non-profit. Now as project manager she knew much more about the project than he did.

“Yes Bo, always busy, but what can I do for you?”

Bo looked a little sheepish, she had something to say but knew it wouldn’t go over well.

“There are a couple of dashboard developers here. They’re close friends with my best project assistant and have a few ideas on how we can improve our data collection and analysis.”

Evaluation again.

Didn’t Bo see the stack of papers on Evan’s desk? She knew the deadlines they were up against better than anyone. Who has time for evaluation!

Bo didn’t wait for an answer, seeing Evan’s face she quickly changed the topic.

“Justine Marley’s also here. She said she knows you, should I send her in?”

Evan nodded.

With that Bo turned, closed the door, and headed back out to the main office.

Evan hadn’t seen Justine for years. He met her as he was just getting started on his grant proposal, Justine was at the time in the final year of her grant.

“Evan, I’m here to warn you.”

“Warn me, about what?”

“It’s your funding Evan, you’re at risk of losing it and everything you’ve built here if you don’t take your evaluation more seriously.”

“But it’s only year two, there are still three years, that’s plenty of time to worry about evaluation.”

Justine, nodded.

“I thought that too Evan, but trust me, it’s not as long as you think. And the longer you wait, the harder it gets to actually collect accurate data and analyze it appropriately.”

***

That night as Evan Scott readied himself for bed, he couldn’t help but think about Justine’s warning. “What if she was right?”

“Bah humbug he thought, there’s still time.”

Closing his eyes Evan drifted off to sleep.

***

*Crash*

“What was that,” Evan jumped out of bed.

All of a sudden a figured emerged into his room.

“Hello Evan, I am the spirit of evaluation past. I am here to show you what was.”

With that the spirit whisked Evan away to a moment in time that seemed like ages ago. There he was, sitting at a table alongside Professor Fizziwig.

“Why it’s our initial evaluation planning session, and there is old Professor Fizziwig. He was one of the most influential figures in our initial project design. He helped us develop a plan that was systematic, outcome-based, and measurable.” Evan remembered the wisdom of the old professor fondly.

“You were so filled with hope and optimism at that time in your program’s development. But what happened to that evaluation plan? The moment you hit the least bit of resistance it was abandoned. You didn’t even bother to call Professor Fizziwig to see if he could help you adapt the plan for the new realities.”

“Tonight you will be visited by two more spirits. Listen to them Evan, your program is not yet lost. There is still hope.” And with that the spirit was gone.

***

The next spirit who arrived introduced herself as the spirit of evaluation present.

“Come with me Evan.”

The spirit took Evan to a small board room. Crowded around the table were some of the most important figures in the local community.

In front of the room was a young man, presenting in front of a power point deck.

“Spirit, who is that young man?”

“Why it’s tiny Jim, Bo Cratchet’s favorite project assistant. He has recently taken on the role of evaluator.”

“But what is it that he’s presenting? It’s a small chart with hardly any data?”

“But don’t you remember Evan. You haven’t committed the resources towards any type of evaluation. Poor Evan has meetings like this on a regular basis. Trying to justify the importance of the program without anything to support his claims.”

“He believes in your program Evan. He knows it works, but try telling that to this room. They will only trust the gut of a program assistant for so long.”

***

All of a sudden the spirit of evaluation present was gone.

In walked another spirit, “hello Evan, I am the spirit of evaluation future. Let me show you the mark you’ve made on the community. At least if things keep going as they have.”

The spirit transported Evan to the location of one of his program’s top sites.

But the site was empty, a closed sign hanging on the door.

“Where is everyone spirit?”

“They are gone. Your program lost its funding, there is nothing for them here anymore.”

“Bo Cratchet? Tiny Jim? What has happened to them?”

“Looking for new work I’m afraid. Tiny Jim is considering law school or business school. His time as a project assistant was tough, he’s decided to move away from the nonprofit sector.”

“And the program itself, has it spun off into a new implementation elsewhere?”

“I’m afraid not. There was no good evaluation data to prove that it worked. The lessons learned were inadequately disseminated in long reports that nobody will bother to read. It’s as if your project never existed.”

“No spirit! Say there is still time, we can show the program’s impact.”

We can at the very least share all of the hard lessons we have learned over time!

Please spirit!”

“Oh Evan, the future is always shifting and still very much open for you to alter.”

Luckily it was all just a dream.