The May 13 Group

the next day for evaluation

You are here: Home / Archives for cplysy

cplysy

Comment on Evaluation is Not Applied Research by Kerry Keating

Hello Dr. Wanzer,

I am a Master’s in Education student currently studying the difference between research and evaluation. I had always struggled grasping the differences in the two, and felt they were just carbon copies of each other, just under different names. This post, especially the hour glass illustration, helps me clearly identify the differences and will be using it in the future to help define research and evaluation correctly.

One question I have concerns the ‘Areas of Difference’ table: Under ‘value judgement’, it states that research is neutral in this aspect. Could you argue that this would depend on which paradigm is used to conduct the research?

Thank you,
Kerry

Thinking Frameworks

You might have heard or read about concepts like Systems Thinking and Design Thinking (both with and without capital letters) and asked yourself: what do they mean?

We see both of these are frameworks for thinking about problems. It’s somewhat confusing, but both systems thinking and design thinking are more than just ways of thinking, they often refer to a constellation of methods, tools, and approaches to problems.

For that reason, we tend to prefer using the terms systems practice and design instead. However, as frameworks for understanding problems, situations, and issues we see much benefit in using the term thinking.

Just like the image above: our thinking frameworks can help us determine whether we’re looking at the land, the sea, or both, together.

Why Thinking Frameworks?

Our mindset — the mental models, habits of mind, or ways of thinking about something — is the primary factor influencing what we do (or do not do). It shapes how we see the world around us, what we attend to, and determines what has value. When we become aware of how we think we reveal the biases (everyone has them) that direct our attention so we’re better able to direct them to where we want intentionally.

We use thinking frameworks in our work by starting out with identifying what kind of ones our clients are using. Ask yourself: what central ideas are useful to us in doing our best work?

This might include concepts such as: evidence-based practice, learning organizations, ethics stances (e.g., ‘green’), values-based frameworks, use-centred (e.g., words like ‘practical’ and ‘user-centred’). Any of these provide guides to what is valued in an organization. There are many more of these.

Exploring Thinking Frameworks

Once we’ve done that, we start to interrogate it (see what we mean by that narrative here). This process involves asking questions that connect what someone says, what they do, and what they accomplish. This helps to see where there might be alignment or misalignment.

In dynamic markets or communities it’s easy to see how an organization can be misaligned. Policies, strategies, and organizational practices are designed for a certain time and place using a certain kind of thinking framework and as things change so does the potential utility of what we’ve created.

By identifying how we think, we are better able to determine the benefits of it and make modifications.

A great tool is using visual thinking and simple sketch notes to illustrate our thinking. By visualizing what we think about we can better tell how we think.

Tools like the Cynefin Framework can also focus our thinking (in this case about systems) to help refine our mental models.

Don’t make this complicated. There’s no need to worry about coming up with the correct terms, language or model for how or what you think about. The key is to simply identify and become more acquainted with how you think, see the benefits that confers, and understand its limitations. By seeing the blind spots, you’re better at seeing opportunities.

Cense helps our clients see things differently so they can do things differently. If you want help seeing or doing things differently, reach out and let’s talk about how we can be of service.

Image Credit: Xhulio Selenica on Unsplash

The post Thinking Frameworks appeared first on Cense Ltd. .

Stop multitasking your report writing.

So I have to admit something. I am HORRIBLE at multitasking.

For example, I can’t look something up on my phone while having a conversation. I can’t cook dinner while putting together a grocery list.

“How about listen to your wife while doing anything else?” my wife added after I had asked her for examples.

But the truth is, I’m not alone in this bad at multitasking thing. Most people are really bad at multitasking.

…individuals almost always take longer to complete a task and do so with more errors when switching between tasks than when they stay with one task.”

Multicosts of Multitasking – Cerebrum

So for my report design, I just wanted it to scream, "I was created in Microsoft Word!"

So what does multitasking have to do with report writing?

If it truly was just report writing, there wouldn’t really be an issue. But modern report writing isn’t just report writing.

Creating a modern visual report requires a mix of at least three major tasks:

  • Writing
  • Illustrating
  • Designing

One of the biggest mistakes that can slow down a reporting process is trying to accomplish all three of these major tasks at the same time.

You can complain all you want about Microsoft Word, but you were the one who decided to spend 30 minutes tweaking the design of a table that will eventually need to be recreated in Adobe anyway. You were also the one who spent a bunch of time creating a chart in Excel that will need to be redesigned with new brand colors and fonts.

Cartoon showing social media messing with the passage of time.

How to single task your report writing.

The solution to the multitasking issue is to intentionally single task your reporting process. This means separating out the different report creation tasks.

Let’s talk about each one of the big three individually.

  1. The design task.
  2. The writing task.
  3. The illustration task.

1. The design task.

So I listed the design task first. This was intentional.

Lots of evaluators tend to think about design as being something you do after you finish your report. But this can be a pretty big mistake.

Here is what happens. Evaluator sits down to write report, opens up Word. Proceeds to brain dump into word document. When nearing end of draft 1, decides that nobody would likely want to read their stream of thought 200 pages.

Okay, so maybe that’s an exaggeration (I hope). But you get the point. One of the things that happens when you just write first is that you tend to over do it. You write more than you need, and sometimes WAY more than you need.

Even before you put pen to paper you know certain things that will influence the design of the report.

Such as…

  • Target page count.
  • How visual you want the report.
  • The branding guidelines for your organization.
  • The sections that need to be included.

Take all of that and design a report shell with the proper fonts and colors. You can change this later, but having it will show you how many words and pictures you’ll need. It will also give you a sense of what you can’t fit, and save you from writing things that will definitely get cut.

And if you don’t know the proper fonts, colors, and other branding guidelines, now is the time to find out.

2. The writing task.

Now that you know where you’re headed, it’s time to start writing.

This is the time to open up Word or Google Docs. Don’t worry about formatting, charts, or pictures. Just the words. Also make sure to use headings and subheadings where appropriate. And if you have something you want in a call-out box or sidebar, don’t bother creating the call-out box or sidebar. Just write something like “Sidebar:” as a header for the text.

And what about data tables? I would suggest putting in the data tables, but not worrying about the formatting. As long as it is readable. Often it can be really helpful to see the numbers you are discussing during the writing process.

Eventually you are going to take the words from this document and put them into the report shell. But I suggest waiting until you’ve gone through most, if not all, of your drafts and revisions.

Most of the report was written in the first three days. I spent the rest of the time trying to make table 7 look okay.

3. The illustration task.

So illustration is not the same as design. During the design process you are going to put in placeholder images and charts. During the illustration process, you are going to create your images and charts.

Data visualization is an illustration task. Your goal is to make the data approachable, and you also want it to fit within the context of your report.

Creating explanatory figures and diagrams is an illustration task. Your goal is to create a visual explanation, and fit it within the context of your report.

Deciding on stock images is also an illustration task. The types of visuals you choose should support the words in your report.

Bouncing back and forth.

Okay, so this is almost never entirely chronological. You are likely going to have to task switch throughout your reporting process. But do it intentionally.

I usually start with design then go into writing. After writing I Illustrate. Then I go back into design mode to pull everything together. I also might need to tweak the designs where the content demands a different approach.

If you are working with a team, get feedback at all the different stages. Approve the design shell/template before putting in all the words. Try to get the writing as complete as possible before putting it all into the template (multiple revisions). Same with illustrations.

All the other tasks.

So designing, illustrating, and writing are not the only tasks required when writing a report. You’ll also likely need to incorporate tasks like proofreading, editing, sourcing, accessibility checking, etc. Just like with the other tasks, separate those parts of the process.

How do you create your reports?

  • Do you start with notecards or sticky notes?
  • Do create your design shell first, or just open up Word and start writing?
  • Do you treat illustration as a separate task or just insert images/charts as you write?
  • Do you start by building a detailed outline with bullet points and goals for each section of your report?

Let me know in the comments.

Comment on Dissertation RQ3: How do researchers and evaluators differ in use, interpersonal factors, and research/evaluation factors? by Cristina

Hello Dr. Wanzer. I hope all is well. I came across your website through a YouTube video on research and evaluation on James Pann’s Channel. I am currently a graduate student at ASU and working on a class discussion project that prompted us to reach out to a professional/ author (reliable source) to discuss the research and evaluation in our current course: introduction to research and evaluation in education. The purpose of me writing to you is to gain knowledge and understanding research and evaluation and to apply it to educational enquiries. I would also like to know the tips you suggest to use when conducting research and evaluation in assignments and in an applied project/work? How would you describe the differences between research and evaluation, and the similarities between them. This might include of course the importance of each approach. We are currently focusing on the work and textbook by Mertens (2020). I would be delighted to hear from soon so I can share your thoughts with my class this week on n Sunday.

Thank you,

Cristina

Change Takes Time: How to Practice Patience in Report Redesign Processes

This guest post comes from Abby Henderson. Abby Henderson, MS, is a Project Manager at Veris Insights. Abby got her Master’s in Program Evaluation and Data Analytics from ASU in 2019, while working at the Center for Applied Behavioral Health Policy within the Watts College of Public Service and Community Solutions. In 2021, Abby joined the team at Veris Insights, where the focus is on bettering the world of work through providing first-class service and research on university recruiting and talent acquisition. In her free time, Abby loves to fall down internet rabbit holes about random topics, build slide decks no one ever sees, and take long walks around Phoenix with her dog. You can connect with her on LinkedIn or by sending an email to ahenderson@verisinsights.com.

We’ve all been on the receiving end of a long, boring, text-heavy report. It can be challenging to sit down and read, rarely translates insights into action, and is (for lack of a better word) boring. Well… I used to be an author of reports like that.

The Everything Report

At the start of my career, I thought the most important thing a report could convey was… well… everything. My thinking went something like this: “All of the background information, data, methods, and recommendations needed to be extensively covered.  The more technical and academic the language, the better. If I can impress people with my language and expertise, they’ll be more likely to follow the recommendations I suggest.” However, that line of thinking functions under the assumption that people will read to the end of the report to get to those recommendations.

What I came to realize early on was that people, in fact, did not want to read through a long and technical report to get to the recommendations buried at the end. On top of that, spending so much time writing extensive reports was taking time away from strategic and creative thinking about what the data means. I had so little time left for that type of thinking that my recommendations were often vague, unhelpful, or lacking in creative thinking.

Introduction to Data Visualization

Cut to a session I attended hosted by the Arizona Evaluation Network and led by Deven Wisner and Nicole Huggett on data visualization. My brain lit up with curiosity and joy at the idea that data could be conveyed simply, succinctly, and visually. That session got me started on a new journey and led to me asking new questions.

Instead of asking, “How much information can I fit in this report? How technical can I make my language?” I was now asking questions like, “What is the purpose of this report? How do I hope these findings are used? How can I communicate that clearly, succinctly, and visually?”

The next thing I knew, I was enrolled in Report Redesign through Depict Data Studio and spending much of my free time thinking about data communication. More broadly, I was (and continue to be) interested in how we bridge the gap between technical expertise and lived experience, and how we communicate across that gap regardless of building a bridge.

Starting a New Conversation

I started to have conversations with my colleagues about how we could change our reporting. I started by suggesting we add more data visualizations and fewer tables.

When I met resistance to this idea, I started to produce two versions of our reports. One version included the tables we were accustomed to, and the second version included more elements of data visualization. Through providing both options, I was able to slowly garner traction and buy-in on including data visualizations.

The next proposal I made was to include infographics, one-pagers, or shorter summaries with our reports for individuals who may only be interested in the data from a high level. Again, I took on responsibility to demonstrate what I was envisioning and how I thought it could work. This meant taking on extra workload to create the products our leadership was accustomed to, as well as products I wanted us to explore. I brought up data visualization in meetings, attended webinars, and tried to increase the data literacy on my own team.

I was hopeful but hesitant during this timeframe. After all, change takes time, and change takes even more time when you’re suddenly grappling with a pandemic and an upending of our work lives as we previously knew them to be. In this new normal, I was suddenly surrounded by data visualization. The conversation shifted in the world around us, and therefore the conversation shifted in our office as well. There was no longer a question about the utility of data visualization in reports, as we were seeing firsthand how impactful a good visualization can be to convey a message.

The conversations about change continued, as I slowly began to notice how things were shifting in our offices. Suddenly, others on my team were hoping to collaborate on infographics. I was being asked to take on more projects, but each new project was a step in the right direction. We were no longer producing reports without visualizations, and we were starting to explore including one-pagers with our annual reports as a standard across projects.

Around this time, I ended up in a new position at a new company that uses data visualization as a bedrock of our work and didn’t get to see firsthand how the process in my previous office continued to change. However, I found out from colleagues that they now include an infographic or one-pager as a standard with all annual reports, are exploring ways to add more visual elements to quarterly reports, and are continuing to increase the internal skill around data visualization.

Lessons Learned

Here are three.

Patience

The most important lesson I learned in this process was one of patience. From the moment I made my first data viz, I wanted to change everything immediately. That was neither realistic nor feasible, and looking back I wish I had aimed for a slow burn to change our processes.

Self-Reliance

By deciding this was the hill I wanted to climb, I also had to be prepared to climb alone. Sometimes this included late nights in the office making two versions of the same report. Sometimes (often) this included frustration with my own lack of knowledge about data viz best practices. I had to trust that this process was worth the extra work. However, looking back, I wish I had had better boundaries around taking on extra workload to accomplish this change. Finding other champions on my team earlier on may have helped with that as well, as data visualizations are nearly always improved by collaborative brainstorming.

Mistakes

Here’s the thing: anytime you learn something new, you’ll make mistakes. I made plenty, both in my actual visualizations and in my attempts to create internal change in my organization. The goal is not to make zero mistakes, but to use your mistakes as a jumping off point for new learnings. Accepting that mistakes are part of any change process can provide grace from the start about the challenges you’ll encounter.

en English
af Afrikaanssq Shqipam አማርኛar العربيةhy Հայերենaz Azərbaycan dilieu Euskarabe Беларуская моваbn বাংলাbs Bosanskibg Българскиca Catalàceb Cebuanony Chichewazh-CN 简体中文zh-TW 繁體中文co Corsuhr Hrvatskics Čeština‎da Dansknl Nederlandsen Englisheo Esperantoet Eestitl Filipinofi Suomifr Françaisfy Fryskgl Galegoka ქართულიde Deutschel Ελληνικάgu ગુજરાતીht Kreyol ayisyenha Harshen Hausahaw Ōlelo Hawaiʻiiw עִבְרִיתhi हिन्दीhmn Hmonghu Magyaris Íslenskaig Igboid Bahasa Indonesiaga Gaeilgeit Italianoja 日本語jw Basa Jawakn ಕನ್ನಡkk Қазақ тіліkm ភាសាខ្មែរko 한국어ku كوردی‎ky Кыргызчаlo ພາສາລາວla Latinlv Latviešu valodalt Lietuvių kalbalb Lëtzebuergeschmk Македонски јазикmg Malagasyms Bahasa Melayuml മലയാളംmt Maltesemi Te Reo Māorimr मराठीmn Монголmy ဗမာစာne नेपालीno Norsk bokmålps پښتوfa فارسیpl Polskipt Portuguêspa ਪੰਜਾਬੀro Românăru Русскийsm Samoangd Gàidhligsr Српски језикst Sesothosn Shonasd سنڌيsi සිංහලsk Slovenčinasl Slovenščinaso Afsoomaalies Españolsu Basa Sundasw Kiswahilisv Svenskatg Тоҷикӣta தமிழ்te తెలుగుth ไทยtr Türkçeuk Українськаur اردوuz O‘zbekchavi Tiếng Việtcy Cymraegxh isiXhosayi יידישyo Yorùbázu Zulu