allblogs

Apr 28 2021

Un enfoque para valorar la calidad de las evaluaciones durante el desarrollo

Siguiendo con nuestra sección “Evaluación durante el desarrollo“, ya comentamos en el post “Control de calidad de las evaluaciones durante el desarrollo (o adaptativas)” que la Oficina de Evaluación de UNFPA ha desarrollado un nuevo y relevante “Marco de Garantía de calidad y valoración de las evaluaciones durante el desarrollo (o adaptativas) en UNFPA” para ayudar a la Oficina de evaluación y a terceros a valorar la calidad de las evaluaciones durante el desarrollo (o evaluaciones adaptativas) ejecutadas por/para esta agencia.

En este informe se indica que el enfoque para valorar la calidad de las evaluaciones durante el desarrollo (o adaptativas) se organiza en torno a seis características clave:

1. Una combinación de principios de evaluación durante el desarrollo y normas de evaluación del UNEG

2. Énfasis tanto en (a) informes de evaluación (productos) como (b) proceso de evaluación

3. Calificar la adherencia de la evaluación a las prácticas mínimas para cada principio / estándar.

4. Tres fuentes de datos y métodos

5. Un proceso de cinco pasos

6. Opciones para la escalada, replicación o ampliación

Cada una de estas características clave se describe a continuación:

1 Principios y estándares de evaluación del desarrollo

La manifestación de los principios / estándares de calidad para la evaluación durante el desarrollo es muy sensible al contexto: la eficacia con que los evaluadores y l@s implicad@s, los apliquen depende de una variedad de factores contextuales, incluida la capacidad institucional, la duración de la iniciativa, la naturaleza de la iniciativa que se evalúa, y dinámica de las partes interesadas, entre otros factores.

La valoración de la calidad de las evaluaciones durante el desarrollo se centrará en la medida en que el proceso de evaluación y los productos pudieron emplear características o prácticas mínimas para cada uno de los once principios y estándares de calidad: uno compartido (centrado en la utilidad / utilización), siete que son específicos de la evaluación durante el desarrollo y dos específicos del UNEG.

Principio compartido

1. Evaluación centrada en la utilidad / utilización

Principios de evaluación durante el desarrollo

2. Propósito de desarrollo

3. Rigor de la evaluación

4. Nicho de innovación

5. Perspectivas de la complejidad

6. Pensamiento sistémico

7. Co-creación

8. Comentarios oportunos

Normas UNEG

9. Independencia e imparcialidad

10. Credibilidad

11. Igualdad de género y derechos humanos

2 Productos y procesos de evaluación del desarrollo

En las evaluaciones de desarrollo, la calidad de la evaluación se expresa a través de:

• Productos de evaluación de calidad (por ejemplo, informes formales, resúmenes de aprendizaje en tiempo real, presentaciones de PowerPoint, memorandos de evaluación del desarrollo, notas de retroalimentación) y

• Procesos de evaluación de la calidad (por ejemplo, relaciones, discusiones evaluativas, revisión conjunta de la retroalimentación de la evaluación, toma de decisiones basada en datos).

Un enfoque convencional del control de calidad en las evaluaciones durante el desarrollo que se base completamente en valorar la calidad de los informes finales tradicionales tendría dos consecuencias perversas:

1. Examinaría solo un elemento, a veces muy modesto, de la experiencia de evaluación del desarrollo (es decir, un informe formal). Por lo tanto, perdería importantes interacciones de evaluación del día a día y múltiples formas de proporcionar retroalimentación en tiempo real que comprenden el mínimo de evaluación del desarrollo.

2. Con el tiempo, incentivaría a los evaluadores durante el desarrollo a realizar un esfuerzo desproporcionado en el desarrollo de un “control de calidad” para los evaluadores , mientras que disminuiría drásticamente el valor práctico de la evaluación durante el desarrollo para sus usuarios principales.

Para asegurar una valoración completa y más precisa de la calidad de las evaluaciones durante el desarrollo, el Marco de Calidad de la Oficina de Evaluación de la Organización debería cubrir tanto el producto como el proceso de las evaluaciones durante el desarrollo que se examinan.

3 Calificación de la adherencia a las prácticas mínimas

La valoración de las evaluaciones durante el desarrollo se realiza calificando el grado en que los productos y procesos de evaluación exhiben estándares de calidad para cada uno de los once principios / estándares de calidad. El sistema de calificación podría incluir calificaciones de bueno a débil.

4.Fuentes y métodos de datos

El proceso de evaluación de la calidad de las evaluaciones del desarrollo debe ser realizado por un evaluador externo independiente utilizando tres métodos.

Métodos para evaluar la calidad de las evaluaciones del desarrollo:

1. Evaluación realizada sobre la base de una revisión de productos o informes de evaluación de desarrollo finales y en tiempo real (p. Ej., Memorandos, mazos de presentación, etc.)

2. Evaluación realizada sobre la base de entrevistas con informantes clave con un número selecto de usuarios, evaluadores y comisionados principales previstos

3.Evaluación realizada sobre la base de entrevistas con el personal y los directores de evaluación

5 Los pasos para evaluar la calidad de las evaluaciones durante el desarrollo

La evaluación de la calidad de las evaluaciones de desarrollo de la organización es un proceso que consta de tres pasos principales.

Paso 1: determinación del alcance

Confirmar y finalizar la evaluación de desarrollo que se va a evaluar, las fuentes de datos clave y los contactos (por ejemplo, productos de evaluación, lista de usuarios de evaluación primarios, equipo de evaluación y personal), entregables, hitos clave y presupuesto.

Paso 2: implementación: Completando las siguientes cinco tareas:

Tarea 1: Revisar y calificar los productos de evaluación del desarrollo (por ejemplo, informes, presentaciones)

Tarea 2: Entrevistar y / o administrar encuestas de usuarios

Tarea 3: Entrevistar al evaluador durante el desarrollo y a los gestores de evaluación de la organización

Tarea 4: Elaborar el informe de evaluación de la calidad: Recopilando calificaciones de las tres fuentes y resumiendo el fundamento de cada calificación.

Tarea 5: Presentar el informe de valoración de la calidad

Paso 3: Revisión y aprobación de la Oficina de Evaluación de la Organización

El tiempo requerido para completar los pasos variará para cada evaluación, pero generalmente debe completarse en un mes.

6 Opciones para ampliar la valoración de la calidad

El enfoque para valorar la calidad de las evaluaciones durante el desarrollo describe un nivel mínimo de esfuerzo requerido para obtener una evaluación completa y de alto nivel de la evaluación.

Hay (al menos) dos razones para decidir ampliar, como las siguientes:

1 Para dar cabida a una iniciativa que es inusual por su tamaño, complejidad o duración, y / o un gran número de usuarios y evaluadores de la evaluación.

2 Para abordar los grandes riesgos que implica la evaluación, lo que requiere una revisión más detallada de la calidad de la evaluación durante el desarrollo.

Hay (al menos) cinco formas de ampliar este enfoque:

1. Ampliar la cantidad de usuarios, evaluadores y gerentes de evaluación de organizaciones entrevistados / encuestados.

2. Incluir entrevistas de seguimiento con participantes adicionales seleccionados de la encuesta para explorar temas clave con mayor profundidad.

3. Introducir nuevas fuentes de datos y métodos (por ejemplo, revisiones por pares).

4. Explorar aspectos particulares de la evaluación durante el desarrollo con mayor profundidad (por ejemplo, un incidente crítico, un conjunto de decisiones, un resultado específico).

5. Emprender un proceso de toma de sentido compartido entre las partes interesadas en la evaluación del desarrollo, en lugar de llevarlo a cabo únicamente por el evaluador de la calidad de la evaluación del desarrollo.

L@s gestor@s de evaluación determinarán si la garantía de calidad puede ampliarse y cómo se puede ampliar en consulta con las partes interesadas de la evaluación durante el desarrollo, y reflejarán este compromiso en un presupuesto de evaluación más grande, un cronograma más largo y una persona o empresa para completar la valoración de calidad.

Apr 28 2021

What are Social Objects? [Behind the Cartoon Podcast]

So I decided to try out podcasting. Today’s first ever episode is about Social Objects. Why? Because it basically covers the theory behind my cartooning.

Show Notes:

Just in case the embed above is not working, you can find the podcast by clicking here.
Social Objects: Everything You Ever Wanted to Know! (2007 blog post on gapingvoid.com, updated in 2017)
Why some social network services work and others don’t — Or: the case for object-centered sociality (a 2005 blog post by Jyri Engeström who was the inspiration for Hugh McLeod)

Related Cartoons

freshspectrum cartoon by Chris Lysy. "Can I tweet a link to my published but paywalled article? Or is that against unwritten social media protocol?"

freshspectrum cartoon by Chris Lysy. "We considered doing something creative, like cartoons or infographics. But we value looking professional over reaching an audience."

freshspectrum cartoon by Chris Lysy. "I'm relying on all of you to help share this critical message with the world."
Later that Day
"How was the presentation? Any takeaways?"
"I honestly can't remember."

freshspectrum cartoon by Chris Lysy.
Social network analysts (3 people hanging out by an SNA diagram).
Antisocial Network Analyst (1 person sitting alone at their computer).

Welcome to behind the cartoon.

This is a podcast about the people and ideas that have inspired the cartoons that I draw on my blog at freshspectrum.com.

My name is Chris Lysy.

Today’s question. What are Social Objects?

***

When you think about social networks, what pops into your mind?

You probably see one of those little diagrams with little dots or nodes and connecting lines. Or maybe even little people icons with lines drawn between them.

Usually we think about these networks as people connecting to other people.

But what this vision misses is the way in which people also connect to objects.

Often, it’s their connection with those objects that connect them to other people.

Think about it.

Let’s say you have a group of friends you met when you joined a softball league. The people on the team all became part of your friend network. But it was the team, and the league, that facilitated the network’s creation.

There are lots of interesting people with interesting ideas. But it takes a person putting those ideas into some type of object before most other people will discover them. Books, journal articles, podcasts, blogs, and presentations are all different types of social objects.

And yes, my cartoons are most certainly designed to be social objects.

I learned about the concept from a cartoonist by the name of Hugh Mcleod who goes into a good bit of detail on his blog at gapingvoid.com. I’ll share one of his blog posts in the show notes.

Hugh applied the idea of social objects to his own cartooning process, focused mostly around business topics.

And that’s also what I try to do, but with academic topics and concepts.

Most academic fields and subjects are, in my opinion, under illustrated. When most work is shared behind closed doors in conference presentations and through journal articles, illustration is only really quasi-important.

But on the web, things are different. The digital world is a visual world. The inability to depict complex or complicated subjects is problematic, because it limits the spread of ideas. Often incredibly important ideas.

So I draw cartoons, attempting to illustrate important ideas. And as a cartoon, that idea spreads much further than it would without the cartoon. When a cartoon works, you find it in social media posts, blog posts, and presentations. It becomes an object with an orbit and connects people in a way that no journal article ever will.

Now that’s just what I’m doing with the concept.

There are all sorts of social objects you can create to spread your ideas. Infographics, podcasts, and webinars all travel pretty well digitally.

What kind of object can you create to help your ideas spread further?

***

Thanks for listening. Before you go about your day, check out the description for a link to today’s show notes. In addition to links that include more info on social objects, you’ll also find a few new cartoons inspired by this topic.

Please enjoy and share. And if this episode inspires you to create your own social object, leave me a comment, I would love to see what you created.

Apr 27 2021

How to Organize Your PowerPoint Slides by Adding Sections

Want to organize your PowerPoint slides a little better? There’s a behind-the-scenes trick that I love using in my own presentations: Sections!

In March 2021, I was speaking with GEDIs about dataviz tricks for presentations. (The GEDI program is the Graduate Education Diversity Internship within the American Evaluation Association.)

In March 2021, Ann Emery spoke to participants in the Graduate Education Diversity Internship program within the American Evaluation Association.

I was scrolling through my slides, and someone asked how I created these “sections” to organize my content.

Adding sections to PowerPoint has been a gamechanger for me personally. They help me stay organized behind the scenes, which helps my audience, too. An organized presenter = an organized presentation = a happy audience that can learn from us headache-free.

Watch the 7-Minute Tutorial on Sections

I started to write a blog post with screenshots about sections… but that felt impossible. I wanted to show you how sections work, so I recorded you a tutorial.

What’s inside:

A demo of what sections are
How to add them
How I use them to hide topics
How I use them to re-order topics

What Are PowerPoint Sections?

Sections help us organize our slides into meaningful groups, categories, chunks, buckets, or chapters of a presentation.

In the video, you’ll see what they look like.

Can you spot the words above some of the slides?

Sections help us organize our slides into meaningful groups, categories, chunks, buckets, or chapters of a presentation. The audience won't see them, they're just for us presenters.

Our audience won’t see the sections. They’re for us, the presenters.

How to Add Sections

It’s easy to add new sections to a presentation.

Here’s how:

I like to be in Slide Sorter view first. (The birds-eye-view of the presentation where you can view all the miniature slides.)
Decide which slide is going to start the new section. Click on that slide to activate it.
Right-click and Add Section.
In the pop-up window, give your section a name. I use names like Introduction, Case Study, Conclusion, etc.

That’s it!!!

Adding sections is easy. Decide which slide is going to start the new section, click on that slide, right-click and Add Section.

How to Use Sections to Hide Topics

I like to keep all my slides for a given workshop within a single file.

While prepping for upcoming talks, I go through the full file – all 900+ slides! – and choose which sections I’ll talk about.

I might cover Sections A, B, and C for one group.

I might cover Sections A, D, and E for another group.

I might add Sections F, G, and H as brand new topics for another group.

(For private trainings, I review the group’s materials ahead of time and hold some planning calls to figure out which sections are going to be the best use of our time together.)

As I’m deciding which sections to include, I simply hide and unhide the slides. In the video, you’ll learn how to hide and unhide slides.

While prepping for upcoming talks, I go through the full file – all 900+ slides! – and choose which sections I’ll talk about by simply hiding and unhiding sections.

(Yes, you can hide and unhide slides without having any sections. I personally like using sections so I can think about an entire category of slides that should be shown or hidden.)

How to Use Sections to Re-Order Topics

I definitely recommend outlining your presentation in a document or spreadsheet before making all your slides.

Our outlines don’t need to be 100% finished before we make our slides. I almost always make changes along the way.

One common change is re-ordering topics. I might envision covering topics A, B, and C, in that order. But later, as I’m designing the slides, I might decide to cover topics A, C, and B.

In the video, you’ll learn how to drag and drop entire sections to re-order them.

I definitely recommend outlining your presentation in a document or spreadsheet before making all your slides. You can then drag and drop them to re-order.

Your Turn

Let me know how you’ve used sections to keep your content organized!

Apr 27 2021

Bundles and Stacks

Two powerful approaches to creating healthy habits are to bundle or stack behaviours together, by design. If we can learn to connect what we do in small ways to other things we can make big changes more easily.

For innovators seeking and making change, these two approaches can help break down a big goal into small, manageable actions.

Bundles

A bundle is a way of tying together two behaviours that don’t normally go together for positive benefit. It takes something you dislike and pairing it with something you like. For example, imagine doing uncomfortable rehab exercises while watching your favourite TV show. Maybe listen to your favourite music while doing your taxes.

There are many activities in our work that are tedious, unpleasant, or undesirable to do. By bundling those with something you do enjoy it can make those unpleasant tasks easier to do.

Consider the barriers to your service that might be unavoidable: how can you bundle those with something positive? How can you make the worst, but necessary, aspects of what you offer, better?

By considering a bundle, we can design for positive outcomes in spite of issues we cannot design around. This is particularly relevant for healthcare (e.g., dental visits, medication taking) or activities requiring paperwork or queuing.

Stacks

A stack is pairing small activities together to produce a larger effect. For example, consider listening to podcasts or e-books while running. This allows you to learn and exercise at the same time.

Another example is drinking a glass of water every time you check your email. This one accomplishes the goal of keeping you hydrated while ensuring you don’t check your email too often (preventing frequent trips to the bathroom).

We can design our ‘stacks’. Practically, this means creating an inventory of jobs to be done and pairing them together when possible. By simply knowing what has to be done we can create stacks that get tasks done together.

Design Considerations

The key to both activities is to break down what has to be done and mapping it out. This can be done using simple paper, sticky notes, a whiteboard, or tools like Miro or Mural. This activity has the added benefit of articulating what has to be done. It is a useful awareness-building exercise to reveal all the tasks and subtasks involved in our work.

Once completed, consider pairing tasks and activities together. What pairs might work? What systems can we put in place to help make things easier?

After pairing, the next step is to prototype and evaluate Try things out. If you wish to drink water with email, consider setting a glass or your favourite mug beside your computer. Create the prompts to remind you to try things out. These are new behaviours so it may take some time to get it right. If, after many attempts, the behaviour hasn’t changed (the bundle didn’t take, the stack didn’t work), try a different one.

The key is to keep trying. Whether this is for individuals or organizations, the key is to keep trying and learning. The more attempts you make, the more you will learn and the more likely you will succeed.

This behavioural strategy is both simple and easy to try and test even if changes are often difficult. If you need help in making these changes and designing for them, contact us – this is what we do.

The post Bundles and Stacks appeared first on Cense.

Apr 26 2021

Join us for the Saint Louis Art Museum’s Virtual Summit on Diversity

RK&A was honored to work with the Saint Louis Art Museum in 2019 to evaluate their longstanding Romare Bearden Graduate Museum Fellowship, which prepares outstanding museum professionals from historically underrepresented groups (particularly people of color) to become leaders in the field. The Bearden Fellowship was one of the first diversity-focused fellowships in the museum world—established long before the recent wave of diversity, equity, access, and inclusion (DEAI) in museums—and ours was the first formal evaluation of the fellowship since it began in 1991. Our evaluation focused on learning from Bearden fellows’ and SLAM staffs’ experiences to not only measure the impact of the Bearden Fellowship, but also to develop a case study with insights and considerations for other museums interested in starting and sustaining a diversity-focused fellowship program.

On May 6, the Saint Louis Art Museum is hosting a free Virtual Summit on Diversity that will convene emerging and established arts professionals from across the country (including many past Bearden Fellows) to discuss ways to diversify the pipeline of museum professionals. RK&A’s Director Stephanie Downey and SLAM’s Chief Diversity Officer and fellowship supervisor Renée Brummell Franklin will discuss the case study we developed with SLAM in the session “Change from Within: A Case Study of the Romare Bearden Graduate Museum Fellowship.” Please register in advance if you would like to attend!

The case study—Advancing Change: A Case Study of the Romare Bearden Graduate Museum Fellowship—will be available free for download immediately following the summit. We are proud to contribute to ongoing dialogue, learning, and action toward creating a more diverse, equitable, and inclusive museum workforce.

Stay tuned for a link to download Advancing Change: A Case Study of the Romare Bearden Graduate Museum Fellowship as soon as it is available!

The post Join us for the Saint Louis Art Museum’s Virtual Summit on Diversity appeared first on RK&A.