allblogs

Feb 03 2021

Evaluating Learning for Innovation

Learning is at the heart of innovation. Our definition of innovation is contingent upon it:

Learning transformed into value, by design

Traditional approaches to learning tend to focus on two things: recall and performance. Thus, we look at knowledge-based tests (consider most of what we took in grade school and college or university) or performance assessments (reviewed accomplishment on a specific task or problem set).

What is often absent from both of these is a consideration of the context and the learner themselves. It’s one thing to be exposed to information or demonstrations of how to do something, yet it’s entirely another thing to actually learn it.

Assessment of Learning: Questions to Ask

Our assessment of learning is inspired by this approach and informs the questions we ask about learning:

Exposure. What level of exposure did we have to the material or experience? How did this exposure treat our sense of understanding of the problem and the context? What salient information did we attend to during this exposure? How (or were) we attentive to what was presented to us and what was the context in which this was presented?

Connection. How is what we are exposed to related to what we’ve already experienced? In what ways does this new material fit with what we have previously learned? Where does this material or experience fit in shaping my goals, intentions, or general direction for myself or my organization?

Reinforcement. What activities allow me to apply what I’ve been exposed to in new ways and in practical ways repeatedly (including fitting with what I already know, whether a theory or practice-focused material)? What systems are in place to allow me to integrate what I’ve been exposed to into my current understanding of the situation? What activities have been set out to enable recall and re-visiting materials in the near and longer-term?

Application. What opportunities do I have to apply what I’ve learned into shaping a new understanding of a problem or steps toward addressing it? What opportunities are present to reflect and integrate this application of knowledge or skill into my existing repertoire of practice or understanding?

Without these, there is a decent chance we will undergo some form of performative learning — doing activities that look like learning but aren’t really effective. Taking workshops on topics that are not relevant to your work or set up too distant from the problems you’re seeking to tackle is one way. Filling your days with reading articles, watching webinars, taking courses, listening to podcasts, and viewing videos on some material without a means to organize and integrate that into your experience.

You will retain some, but probably not enough to warrant your time and energy. Furthermore, this kind of ‘filling’ of your attention actually serves to decrease the likelihood that you’re able to use this because of the sapping of energy from your brain. The more we expose ourselves to, the more energy is required and eventually we’ll be depleted enough that things won’t stick.

Ask yourself these questions above next time you’re looking to really, truly learn as an individual, team, or organization.

If you’re looking to learn more, do more with what you have, please contact us and we’d be happy to discuss ways we can help you be a better learner and innovator.

Better learning comes from better systems, by design. we can help you build them.

Feb 01 2021

Una conceptualización sistémica de la colaboración

Fuente: rethinkingresearchpartnerships

En la línea de nuestras serie sobre “colaboración” e “impacto colectivo“, encontramos el artículo “Más allá de las asociaciones: adoptando la complejidad para comprender y mejorar la colaboración en investigación para el desarrollo global“ de Jude Fransman, Budd Hall, Rachel Hayman, Pradeep Narayanan, Kate Newman & Rajesh Tandon, publicado el 29 de enero de 2021, trata de “Repensar la investigación colaborativa” (Rethinking Research Collaborative/RRC) y sus ocho principios para asociaciones de investigación justas y equitativas, utilizando el concepto de “Pensamiento sistémico y teoría de la complejidad“, que ofrece un marco sistémico para explorar la colaboración en investigación y una propuesta para Mirar hacia adelante: de asociaciones eficientes a colaboración justa.

Nos indican en el artículo que si bien existe una creciente literatura sobre los beneficios de la colaboración en la investigación para el desarrollo, tiende a promover la idea de la “asociación” como un sitio delimitado en el que se pueden realizar intervenciones para mejorar la práctica colaborativa. Este artículo se basa en la teoría de la complejidad y el pensamiento sistémico para argumentar que tal suposición es problemática, ya que separa la colaboración de sistemas más amplios de investigación y práctica. En cambio, el artículo propone un marco sistémico para comprender y evaluar la colaboración. Este marco se utiliza para reflexionar sobre un conjunto de principios para una colaboración de investigación justa y equitativa que surgió de un programa de investigación estratégica y fortalecimiento de capacidades realizado por “Rethinking Research Collaborative” (RRC) para el principal financiador de la investigación del Reino Unido: UK Research & Innovation (UKRI). El artículo concluye que una conceptualización sistémica de la colaboración responde mejor que un enfoque de “asociación”, tanto a los principios de justicia y equidad como a futuros inciertos.

Jan 31 2021

Teoría del Cambio y Gestión Adaptativa: ¿la canción lógica?

Fuente: theoryofchange

Siguiendo con nuestra serie sobre Teoría del Cambio (TdC), apuntamos algunos retos para utilizar la teoría del cambio más allá del diseño de las intervenciones y en apoyo de una gestión adaptativa:

Confirmemos la plausibilidad de nuestra TdC: Después del diálogo interno y el diseño participativo de la TdC, debemos verificar si nuestra “teoría” es plausible.
Desarrollemos el diagrama TdC en el nivel correcto para analizar el progreso de la cadena lógica
En el trabajo de formulación de políticas, se necesitarán múltiples TdC para abordar los cambios deseados en varios niveles. La TdC también debe identificar y abordar los resultados intermedios.
Crear, mejorar y adaptar en el proceso de ejecución y seguimiento, para conseguir una teoría que sea medible.
Una vez diseñada, usemos y adaptemos la TdC durante la fase de seguimiento (a no ser que seamos “mag@s”, desde el diseño no lo sabemos todo, ni hemos resuelto todo): usemos sistemáticamente la teoría del cambio para (1) determinar qué datos recopilar, (2) guiar el análisis o (3) adaptar la ejecución y la propia TdC.
Aspiremos a crear un espejo/observatorio en lugar de un objetivo: La TdC articula de qué quiere la organización rendir cuentas/ser responsable y trabaja hacia atrás para identificar las actividades, estrategias, recursos, capacidades, cultura, etc. El trabajo con la TdC nos lleva a proponer cambios en estos elementos
Hemos de tener en cuenta el contexto externo: una comprensión profunda del contexto externo durante el proceso de ejecución nos ayudará a crear/adaptar para una teoría del cambio más realista.

Jan 29 2021

El marco de competencias en evaluación de Naciones Unidas

Fuente: onboarding-process-header

¿Qué son las competencias?

Las competencias pueden definirse como “grupos de conocimientos, habilidades, atributos y otros requisitos relacionados necesarios para un desempeño laboral exitoso”. Cada puesto de trabajo tiene su propio conjunto de competencias. Al identificar los requisitos de competencia, es importante que el grado de dominio esté relacionado con el nivel del puesto en términos de nivel de responsabilidad, complejidad y autonomía (es decir, no es realista esperar que los oficiales junior y senior posean el mismo nivel de competencias).

El marco de competencias de evaluación de Naciones Unidas (2016) asume que la función principal de un/a evaluador/a es producir evaluaciones de alta calidad que marcan la diferencia. También asume que los jefes de unidad de evaluación y los gestores de evaluaciones contribuyen a estas evaluaciones. El marco proporciona una descripción general de las competencias básicas en evaluación, teniendo en cuenta que l@s evaluador@s, como profesionales, tienen una mayor responsabilidad para marcar la diferencia.

Las competencias están asociadas con los cinco dominios:

• Fundaciones profesionales

• Habilidades de evaluación técnica

• Habilidades de gestión

• Habilidades interpersonales

• Promoción de una cultura de aprendizaje para la evaluación

Jan 29 2021

What is Developmental Evaluation? Cartoon Glossary

This is a series of posts providing quick of overviews of important topics in research and evaluation. Each post in this series will include at least 3 cartoons from my archives and at least 3 links to recommended resources. I only give quotes here and recommend that you follow the links below each quote for more detailed information.

What is Developmental Evaluation Cartoon by Chris Lysy of Freshspectrum.

"As an evaluator I can help improve your program.

But we don't have a program yet. Just a problem, a few ideas, and a little money."

Developmental evaluation supports innovation and development, especially in complex and dynamic situations.

Accountability evaluation, formative evaluation, and summative evaluation all depend on there being a program or model to evaluate. Developmental evaluation is relevant when the program or model is still being created and those involved are figuring out what they want to do and how to do it. Thus, developmental evaluation helps people developing new initiatives to get ongoing, real-time feedback about what is emerging and its implications for making a difference.

Evaluation Flashcards developed by Michael Quinn Patton for the Otto Bremer Trust

What is Developmental Evaluation Cartoon by Chris Lysy of Freshspectrum.

"Formative, Summative, is that all you evaluators got?

Umm, well, umm how about dev-el-op-men-tal."

On a cold Minnesota morning in February, I met with them after two and a half years and said, “You folks have been a great group to work with, you’ve been open to feedback, you’ve made changes, you’ve really adapted, but now we’re moving into the summative period, where we have to decide if the model that has been developed works, and so you can’t make any more changes in the program, because if you keep changing, we can’t answer the question of did it work. It’s got to be stable, standardized, fixed, and that’s now the challenge. So change is done, next two and a half years, everybody
gets the same intervention and then we’ll follow up, and see what’s happened to participants, what they’re doing in their communities, what kind of differences they’re making, how they’re communities view [them].”

The director of their program looked at me and he said, “But we don’t want to stop changing the program.” I said, “No, I understand you’ve been really good about changing the program, but we’re now doing what’s called summative evaluation, and that
means you can’t keep changing the program, the formative piece is over. The Board has contracted me to do a summative evaluation to answer the question does it work? There are a lot of people watching what you’re doing. People want to know if they
should emulate this model. That means summative evaluation.” He said, “No, no, no, no, you don’t understand. We understand that we can’t keep the program the same, we need to keep changing the program because the world around us is changing.” Then he
looked at me, fairly hostilely, and he said, “Formative evaluation, summative evaluation, is that all you people have?”

Well, in truth, those have been the dominant paradigms, with an accountability version of summative evaluation, which is a lot of what IEG does. Quite taken back, I said, “Well, I suppose, if you really wanted to, you know, we’d have to renegotiate the contract, but you know if you really wanted to, we could try doing developmental evaluation.” And they said, “What’s that?” I said, “That’s where you keep developing and adapting.” And they said, “That’s what we want to do. How do we do that?” I said, “Well,
we’ll have to figure that out. I’ll get back to you on that.”

Summary of a Workshop High Impact Evaluations Exploring the Potential of Real-Time and Prospective Evaluations A Workshop Conducted by the Independent Evaluation Group. Washington, DC – January 27, 2010

What is Developmental Evaluation Cartoon by Chris Lysy of Freshspectrum.

Why developmental evaluation exists...

"Our program just started we have no idea where to go or how to evaluate.

We'll start by predetermining the program's outcomes using very little evidence then over the years we'll judge you according to that."

Developmental Evaluation (DE) is an evaluation approach that can assist social innovators develop social change initiatives in complex or uncertain environments. DE originators liken their approach to the role of research & development in the private sector product development process because it facilitates real-time, or close to real-time, feedback to program staff thus facilitating a continuous development loop.

Michael Quinn Patton is careful to describe this approach as one choice that is responsive to context. This approach is not intended as the solution to every situation.

Developmental Evaluation – Better Evaluation

allblogs

Assessment of Learning: Questions to Ask

¿Qué son las competencias?

Footer

Follow our Work