freshspectrum

Is your data dashboard useful? How do you know?

Most of the time it’s a question that gets left unanswered. As if the dashboard getting created is enough, and whether or not it works is really not that important.

Perhaps it’s the data idealist in me, but I think we can do better.

• TLDR: I created a free UX Evaluation eBook – you can download it here.

The “How to Evaluate a Dashboard” Question

Up until recently I’ve seen very few organizations really putting any time or effort into evaluating their data products. A dashboard or report is something that often comes at the end of a project. And by that time, everyone is already moving on to the next thing.

That’s starting to change at least. When an organization has had a dashboard live for a few years, or they start building more and more dashboards, that nagging “usefulness” question starts to become more prominent. Especially if a good amount of work is put into to maintaining/updating the tool.

So how do you evaluate a data dashboard?

The Problem with Defaulting to a Survey

The knee jerk response from a lot of evaluators is to send out a survey.

And yes, this will give you some insight if you’re good at writing surveys and know how to get a good response rate. But a survey also has some major drawbacks.

• You don’t want to survey too often, so it’s hard to use in support of new tool development.
• Even with a well developed survey, you don’t get the depth of information you can with qualitative methods.
• Lots of user experience issues can be found through just a few tests. So a survey can be overkill.
• And surveys can be a lot of work. At least if you do it right.

Plus, I think there is a better method.

The Limitations of a Checklist

So another thing to mention would be the checklist approach.

For instance, there is this data visualization checklist designed by Ann K Emery and Stephanie Evergreen.

Now let me go on record to say that it is a good idea to run through a checklist like this one. It can help guide you towards better visualization practice. And that’s a good thing.

But ultimately it’s insufficient.

One of the things I teach a lot in my workshops is that a bad chart with the right data is more useful than an amazing chart with the wrong data.

Checklists can help you towards creating better quality products. But that doesn’t mean the products will be more useful.

What is UX Evaluation?

UX Evaluation is how I refer to the use of user experience design methods for the purpose of evaluating the usefulness of products or programs.

While UX Design is completed during the development process, UX Evaluation can occur before or after a product has been completed. Insights can inform future iterations or entirely new designs.

Long story short, it’s a qualitative approach that I have found to be really useful for evaluating things like data dashboards, reports, and other data products.

UX Evaluation: How to Evaluate Dashboards, Reports, and Data Visualization

Want to know how I would approach UX evaluation for a data dashboard, report, or other type of data visualization product? I wrote up my method in a free eBook you can download.

Working with a group to create a logic model? I created a little activity book to help you on your way.

Prerequisite (Before you Download)

This activity book assumes that you are trying to take action and do something. Or that you’ve already taken action.

If you haven’t, you’ll have to imagine yourself taking action on something to get any use out of this book.

What is in the activity book?

Basically just a series of question based activities. Instead of getting bogged down in jargon, the activities start with simple exercises.

It starts with these.

• What are you trying to do here?
• Name three people who a stake in what you are trying to do.
• What if those three people answered that first question?

Then we dive into thinking through actions and consequences.

• If I do this…this happens.
• From actions to outputs.
• From consequences to outcomes

Finally we pull it all together.

• Assembling a basic logic model.
• Intentionally complicating your logic model.
• Bonus activity: coming up with a counterfactual.

How much and where to download.

The personal non-commercial version is free but tips are appreciated.
(Just put in $0 to download for free, no judgement). There is also a print-away version that requires at least $10.

Here is a button:

Download the Activity Book

If that doesn’t work, you can download it directly from Gumroad. It’s also on my downloads page.

How many can I print?

Totally okay to print the free version for personal use.

If you want to print several for a get together with your small non-profit, also okay to use the non-commercial version.

I know some of you teach classes, feel free to print it for your students (or send them a copy you downloaded) of the non-commercial version.

Do you make money as a consultant or trainer doing this kind of thing? That’s why I created a commercial version. But if you buy that, feel free to print away as much as you would like.

But I just want logic model cartoons…

For that I suggest either buying my book.

Supporting me on Patreon (my Patrons get access to all my cartoons. They will also get the above commercial version activity book downloadable on Patreon for nothing extra).

Or just pulling them from my big post of evaluation cartoons (I have a bunch there).

If you want to be taken seriously as a data professional, there are some things that you just need to know. The difference between a Bar Chart and a Histogram is one of those things.

This post is a little cartoon illustrated explanation of the differences.

In short: the difference between a Bar Graph and a Histogram. Histograms are a bunch of bars connected to each other, visualizing a distribution of a some continuous quantitative variable. Bar graphs (or bar charts) use proportionally sized rectangles to visualize some type of categorical data.

The purpose of a histogram (where you often see histograms).

So I started my career as a researcher, spending a lot of time looking at survey data. One of the first you do after data collection (or really during data collection), is create a report with all the response frequency tables. For the categorical data you would visualize the frequencies with a bar chart. With any quantitative data, you would visualize the frequencies with a histogram.

Histograms let you see the data distribution, and this is one of the first things most researchers will look at when analyzing quantitative data.

Calling a histogram a bar chart (or connecting your bars) is the data person equivalent of using the wrong “there.” Does it really matter that much? I don’t know…but it will draw attention in certain crowds.

Some of the main differences between bar charts and histograms.

• As already discussed, the whole continuous variable vs categorical variable thing.
• The bars touch in histograms.
• You can’t change the sort order with a histogram (or I guess you could, but you shouldn’t).
• Histograms require you to bin your numerical data.

Putting your data into Bins.

Bins are the groupings you put your data into.

It’s kind of like grades in school. Is there a huge cognitive difference between kids born in June and October of the same year? Is it any different from kids born in February and June? I would think not, but at some point people needed to decide the cutoff point for a certain group of kids they wished to educate together. So they split them into bins (in this case they called them grades).

There is no set rule saying “you must bin like THIS!!!!” So a lot of bins are based on the judgement of the data analyst.

Take age for example, it’s a continuous variable. Let’s say you have a community survey and your responses ranged from a 17 year old to a 93 year old. How would you bin it? You could use 5 year bins starting (for example 16-20, 21-25, 26-30…) or 10 year bins (11-20, 21-30, 31-40…).

Histogram and Bar Chart Resources

• Want to create a Histogram in Excel? Here is a short guide from Excel Easy.
• More interested in creating bar charts? I have a post on How to Create Bar Charts in Excel.
• I also have a sister post on How to Create Bar Charts in Adobe XD.
• Don’t like my explanation and want someone else to explain the difference between bar charts and histograms? Storytelling with Data also has a Histogram vs Bar Chart post.
• Want a more comprehensive guide to setting bin sizes? Here is one from Statistics How To.

This week’s post was written by Khalil Bitar. Khalil is an evaluator and M&E specialist who has been a leading voice in the EvalYouth global movement. He was also the founder and first president of the Palestinian Evaluation Association (launched in 2013). I invited Khalil to write a post sharing his views during the conflict and Israeli airstrikes on Gaza in May. These are Khalil’s words, I contributed the cartoons.

Something isn’t right about evaluation that doesn’t put (social) justice in the forefront. Something is also pretty wrong in my view about paying attention to justice and equity —by evaluators— only concerning particular issues and not others. Are we short in the community on both fronts? It seems so. I will explain why I think so in this blog. 

A little bit of context first. Recently, during the Israeli Occupation’s aggression in my country of Palestine, which resulted in the killing of over 280 people (among them 66 children), I noticed a sort of silence toward what was happening among evaluators. I took it to Twitter. I spoke directly to fellow evaluators. This is what I wrote:

This applies to “friends” and colleagues here on Twitter —really disappointed with the #EvalCommunity‘s silence, esp. those who speak ALL THE TIME about SOCIAL JUSTICE!

Selective justice isn’t justice, and selective activism isn’t activism.#Eval #EvalSoWhite #SaveSheikhJarrah pic.twitter.com/KdJyfAzSSM

— Khalil Bitar (@KhalilBitar) May 15, 2021

“...really disappointed with the #EvalCommunity’s silence, esp. those who speak ALL THE TIME about SOCIAL JUSTICE! Selective justice isn’t justice, and selective activism isn’t activism.”

I wrote that because I noticed that many colleagues and friends from the community —who are usually active in speaking up against injustice in other parts of the world— were mostly silent when it came to Palestine. Many of these colleagues are from North America and Europe, where supporting Palestine and the Palestinian cause is sensitive, often due to conflating criticizing Israel with anti-Semitism within the political circles and mass media outlets. Chris reached out to me in the midst of all of this and invited me to write a blog on Freshspectrum, which I love and often read, so I am writing this blog. 

It was powerful —and important— to observe and be part of the activism by members of the international evaluation community during the Black Lives Matter (BLM) marches in the US and many other places around the world last summer after the cruel murder of George Floyd. It mattered because speaking up against injustice in all ways possible, including on social media, raises awareness about these injustices and challenges the status quo. Among many things, it showed how the evaluation community —like many other people and communities around the world— is eager to speak up against injustice and join forces with others in doing so. 

However, I noticed during the aggression on Palestine that very few colleagues in the community were speaking up against injustice in Palestine. I received some very kind private messages. But, in general, only a handful of colleagues seemed to have followed what is going on in Palestine or share about the matter on their own social media pages as they have been doing concerning the BLM movement, climate change, or indigenous people’s rights. 

This was happening while —even in the US— some progressive congresswomen and congressmen —perhaps for the first time— publicly criticized and condemned Israel’s actions in Palestine requesting to halt the US military aid to Israel; demonstrations were taking place in most big cities in the US, Canada, Europe, and other countries in most regions of the world; and social and mass media outlets were saturated with news about the bombardments, ethnic cleansing in Sheikh Jarrah, and assault on activists and journalists. But this aggression, occupation, ethnic cleansing, and apartheid (as categorized by Human Rights Watch and other human rights organizations) also happened with shameful political support and military assistance to Israel, especially from the US, Canada, and several countries in Western Europe.  

I believe that evaluation colleagues in these very countries have a particular responsibility to speak up against injustice in Palestine. They should speak up against injustice in other places too, especially where their governments have been destabilizing other countries’ political, economic, and social systems —in Asia, the Middle East, Africa, Latin America, and other places. Evaluators can’t separate themselves from what is going on around the world. They have even greater responsibility and duty when their governments actively fund and defend a country colonizing other people, ethically cleansing them, and exercising apartheid, like in the case of Palestine. The US alone provides Israel with $3.8bn annually in military aid. 

I understand that many may believe that the situation is complex and that they don’t understand it. It is not. It is an occupation, apartheid, and ethnic cleansing. One of the Occupation’s —and its enablers’— main tactics has been to make the situation seem so complex so that settler colonialism continues as it is. Others might be concerned that supporting the Palestinian people’s struggle for liberation could be seen as anti-Semitic. It is not. They can check the work of the many Jewish organizations and activists who are among the most outspoken against the Occupation and Apartheid, or the work of other Palestine activists who —in words and deeds— fight anti-Semitism hand in hand with their struggle against the Occupation. 

Others might feel the injustice in Palestine is not relevant to them or the issues on which they focus. To those colleagues, I find nothing more relevant than what Martin Luther King Jr. once said: “Injustice anywhere is a threat to justice everywhere.”

Dear Evaluator, I think you should blog.

Partially for selfish reasons.

The social world is undergoing fundamental changes that will shape our collective future.

As an evaluator you are positioned to observe those changes first hand. And I want to hear what you have to say. We all have unique perspectives and backgrounds, but it’s harder to hear you when you don’t speak.

But beyond that, blogging has been fundamental in guiding the path of my own career. As it has for some of my friends and colleagues. And I want that for you too.

In today’s post.

• A little blogging advice…again?
• Showcasing expertise in the digital era.
• Blog the job you want, not the job you have.
• Blogging as a resume booster.
• Blogging as a network builder.
• Blogging away the Google gaps.
• Where are we now? Blogging tech in 2021.
• Get an audience jumpstart by adding your evaluation blog to Eval Central.
• This weeks’ awkward evaluation networking session with Isaac Castillo.

A Little Blogging Advice…Again?

This is certainly something I’ve talked about before. Over the last decade I’ve delivered conference presentations and written articles about the subject.

In 2013 I wrote a post pulling together blogging advice for researchers and evaluators from a group of 22 bloggers. Some are still blogging today on the same domains, some are blogging on new sites, and a few have stopped completely.

That post was this site’s most popular post for years. And I think the advice still rings true. It’s not 2013 anymore, there are more people sharing their views on the web. But don’t think for a second that your voice is not needed or that a blog can’t help you move your career forward.

Showcasing Expertise in the Digital Era

All the world’s a stage,
And all the men and women merely players;

Opening lines from Shakespeare’s As You Like It

Fair or not, our professional first impressions these days are guided by what comes up on the other side of a Google search.

Your name could be on a series of important journal articles or listed as presenter alongside other big names at prestigious conferences. But if that stuff isn’t shared well on the web (associations and the academic publishing industry are not known for their web savvy) your reach will have its limits.

In the digital era, a blog is an amplifier. It can project your voice beyond the limits of the ivory tower. It’s the quickest way I know to change what people find when they search for you.

Blog the Job You Want, Not the Job You Have

On Wednesday of this week I’ll be chatting with Isaac Castillo about getting real world experience as an evaluator. It was that upcoming conversation that inspired me to write this post.

I hear from a lot of evaluators who want to know how to practice something like data visualization when you don’t have the opportunities at work. And my response is almost always to start a blog.

When I started blogging I was writing about new techniques, software, and visualization before ever having the chance to really apply the lessons through my position. Through much of the beginning parts of my career I would spend most of my time coordinating data collections, programming digital surveys, or following up with non-respondents.

It was only after I had shown what I could do through my blogging, that the creative opportunities started to present themselves.

Starting a blog gives you an opportunity to have something that you control. You get to call the shots on what you work on and what you don’t. Want to get practice creating infographics? Create a series of blog posts. Want to dive into R programming? Create a series of blog posts.

Let the professional you want to be guide your blogging. Don’t lie about your credentials, but don’t limit yourself to sharing only about your current paid work/past experiences.

Blogging as a Resume Booster

Want to show off a skill in your resume that’s not part of your current role?

Blog about it.

There is no reason that the publications section of your resume can’t include blog posts. Use that to your advantage.

When applying for a job, create a series of blog posts that align your experiences with the needs listed in a job posting. Not only will it improve your chances of being interviewed (since the posts will better align your background with the job posting) but it will also help you prep for a possible interview in the future.

Blogging as a Network Builder

The act of blogging can also help you build your network.

Want to reach out to someone you don’t know? Say a company you admire or an expert you would love to meet. Why not ask them if you can do a little interview for a series of Q&A blog posts.

You can bundle multiple interviews together into the same post or just take it one at a time. This also works well for independent consultants. Collaborating on a blog post with a potential client is a good, “non-salesy” way to get your foot in the door.

Blogging Away the Google Gaps

Do you have a specialty or area of expertise in the field?

What happens when you ask Google questions about your subfield? Do you like what you see or do you cringe at the search results?

Honestly, I believe we have an academic responsibility for the search results and

Wikipedia pages in areas for which we claim expertise. I have heard complaints that you shouldn’t trust a Google search to do your research. But complaints alone won’t increase the accessibility of good information.

Experts have agency. Don’t assume someone else will take the time to spread your work if you won’t take the time to spread your work.

Where we are now, blogging tech in 2021

Blogging is easier than ever to get into.

The biggest tech platform for blogging is still WordPress. There are other platforms that let you blog, but WordPress is still definitely the leader.

If you hope to start regularly blogging, I suggest starting with WordPress.com. It’s free, and if you decide in the future you want more control over your site’s design, it’s easy enough to hire a developer and transition to a self-hosted WordPress.org site.

If you just want to write a blog post but don’t want to design a blog. You can do that right now for free using Medium.

You could also write an article using LinkedIn, but I recommend Medium because LinkedIn has a way of locking your post inside its social network. For example, I can integrate an author’s blog posts written in Medium into the EvalCentral feed. I cannot do the same with LinkedIn.

Get an Audience Jumpstart by adding your evaluation blog to Eval Central

I created Eval Central about a decade ago as a tool to help evaluation bloggers reach evaluation audiences.

The site is basically a robot. Here is how it works.

1. A blog gets added to the Eval Central “Feed”
2. The blogger publishes a new post on their blog.
3. Eval Central sees the new post (<1 hour of the original post) and creates a copy, with a link back to the original.
4. The Eval Central version of the post gets automatically shared via social (Twitter, LinkedIn, Facebook). The @evalcentral Twitter account has the largest following among the different networks with over 4,500 followers as of June 2021.

Want your blog added (or want to suggest someone else’s blog) you can do that through my community site.

This weeks’ awkward evaluation networking session with Isaac Castillo

For this week’s awkward evaluation networking (Wednesday at 2 Eastern)I’ll be joined by Isaac Castillo. The starter topic is about getting real world experience as an evaluator. It’s free, and should be a good conversation, hope you can join us.