allblogs

Last month, I had the opportunity to attend the Summer Evaluation Institute hosted by AEA. I met some great people and learned a lot about evaluation. I’m what my family likes to call a “professional student” – I would be in school for the rest of my life if I could. I took pages and pages of notes, organized the resources I was given, and came up with this list of the lessons I learned while there.

1. Evaluation isn’t just for evaluators.

Going into the institute, I was so nervous. With just under a year of professional evaluation experience under my belt, I was sure I would be behind in some of the workshops. That definitely wasn’t the case. Now, I’m not claiming to be an evaluation master, but there were fewer evaluators attending than I anticipated. Project developers and managers, health promotion specialists, policy analysts, and more came from all over the country to learn about evaluation.

This isn’t a new concept for me, but it was strongly reinforced at the Institute. Working with Ann at Community Evaluation Solutions has taught me that building an organization’s evaluation capacity is an important part of being an evaluation consultant. We want to make sure that our clients can and will continue to evaluate their programs, even after our contracts end.

Fun Fact: Ann actually taught a workshop at the Summer Institute this year on this very topic: Tools and Techniques for Assessing and Strengthening Nonprofits’ Evaluation Capacity.

2. The content is the paint, but the presentation is the canvas.

Bear with me. What I mean here is that the canvas is what brings the art together. If you’ve got a ton of information, but no clear way to deliver it, it’s as good as buckets of paint on the ground. The final product is what gets remembered, and if you can’t bring all of your paint together to tell a story, then what’s the point?

Out of the five workshops I attended, there are two that I will likely remember for a while. I am painfully introverted, and as a result of that I have a strong aversion to public speaking. One of the workshops I attended was about strategies to engage your audience when presenting. Not only did I learn about the strategies, but I saw them in practice. The speaker, Sheila Robinson of Greece Central School District, was captivating and informative, which can be difficult. Kylie Hutchinson, the second speaker I will not soon forget, was personable and fun. Usually when a presenter throws facts at you, they are forgotten as soon as you walk out the door. These two speakers presented the information in a way that will make it sticky and easy to recall/repeat. I’m hesitant to say that I am excited to put what I’ve learned to use, because that would mean I have to stand up in front of a group and present, but I already feel more prepared for my next presentation.

3. It really does matter where you work.

As I mentioned in #1, I have been working for Ann for almost a year now. One of the best things about CES is that I’m able to put my hands on so many different projects and learn a little about a lot. Well, the main thing that the Institute taught me is that I haven’t learned a little about a lot, I’ve learned a lot about a lot. I honestly did not expect to learn as much as I did in my first year, and I did not realize how much I had learned until last month.

In my last workshop, we were split up into groups and given a few tasks. In my group there were a couple of young evaluators, an epidemiologist, a program developer, and a few disease prevention specialists. Working through the tasks made me realize how much I already knew about evaluation. I could answer questions that my group had, give examples of work we had done at CES, and explain why certain techniques were used in different situations. I’ve never been very good at networking but being knowledgeable about evaluation helped in that other people wanted to start conversations with me about my experience at CES. Knowledge is power, and Ann sure has given me a lot of it.

Overall, the Summer Institute was an amazing learning and networking opportunity. There are classes for all levels, and I really think non-evaluators would benefit from attending. Hopefully I am able to attend next year, but until then I will learn every day with Ann.

The AEA Summer Institute: A First-Timer’s Impression Daniel Snook is currently working at Community Evaluation Solutions doing a practicum in program evaluation.

My first trip to the American Evaluation Association’s (AEA) summer institute was eye-opening. I was aware of evaluation as a practice and as a useful tool for program development and improvement before I arrived, but I was not aware of the full breadth of Evaluation with a capital ‘E’ (i.e., evaluation as a field). Suffice to say, I now know just how much I don’t know about the incredibly multi-faceted field of evaluation.

My background is in psychology, specifically I’m a PhD student studying Community Psychology at Georgia State University. At the beginning of the conference I was feeling a bit like an incognito psychologist—I didn’t want anyone to realize that I hadn’t been doing evaluation work for years and that I didn’t identify per se as an ‘Evaluator’, at least, not yet. Community Psychology is essentially the study of how communities impact individual, and most programs (including my own) provide some training in social program evaluation. However, learning about a suite of techniques in theory and learning about them in practice are very different things. Occasionally the academic world and the real world come crashing together– that was my experience at the AEA summer institute. It was simultaneously awesome and discomfiting, and it was a great place to learn a lot very quickly! Here are some highlights of what I learned:

1. Theory matters– no matter how applied your work is.

I came into the first session I attended, Program Theory, led by Dr. Stuart Donaldson thinking I would find myself in comfortable (read: ‘academic’) territory. One of the first activities we did challenged that idea almost immediately. Dr. Donaldson asked each of the groups of audience members to evaluate the room in which we were sitting. Each table and its members were then asked to assume roles, for instance, as teams of interior decorators, information technology specialists, or, in our case, firefighters. The number of different ideas about what makes a room ‘good’ or ‘successful’ from these various perspectives was staggering. As you might expect, the lesson became quite clear: your approach to and expectations of a situation (i.e. theory) significantly impacts your practice, even if you don’t explicitly realize it.

2. Avoid “The Curse of Knowledge.”

In her keynote presentation, veteran evaluator Kylie Hutchinson described some of the basics of effectively communicating evaluation results to stakeholders. A prime mistake, she says, that presenters make, whether in evaluation or otherwise, is assuming their audience knows what they know. This is, of course, more of an implicit than explicit assumption; evaluators are consciously aware that their clients do not know every detail of the evaluation, yet their presentations often don’t reflect that. Understanding your audience’s starting point by doing things as simple as laying off the jargon or explaining acronyms can keep your audience from zoning out or rolling their eyes. It’s also tempting to include every detail about your evaluation just because each detail is important to you. But your audience hasn’t been working on the evaluation at that level of detail, so spare them all of your knowledge and focus only on what you know is important to them. If you can say it more simply, then do so.

3. Good presentations are good; bad presentations are terrible.

The AEA summer institute left me with the distinct impression that evaluators are very good presenters. They’re practically oriented, which means they want to get to the bottom of what makes a program successful (or unsuccessful) and then get to the point in telling the client about it. Evaluators also have a very high bar for data visualization, tools for incorporating theory into practice (e.g., logic models), and techniques for transforming vague goals into tangible and measurable specifics. However, I also learned a bit about how NOT to present. I’ve mentioned jargon once already, but I have to reiterate that purposefully using jargon, whether it’s to obscure the fact that you’ve got nothing substantive to say or to create a veneer of professionalism, wastes everyone’s time. Finally, to be frank, not all content is worth presenting. If you’re on the fence about whether the ideas in your presentation are worth sharing, they’re probably not.

4. Invest in your evaluations early.

The readiness is all, and that holds especially true for evaluation. Several of the sessions I attended discussed the importance of being prepared for your evaluation from start to finish to set yourself up for success. Sheila Robinson’s excellent session on strategies for evaluation planning encouraged me to ask the right questions– the why, what, and how of conducting an evaluation– early on (i.e., before beginning an evaluation). In other sessions, presenters made it clear that whether you’re using pilot studies or cognitive interviewing, it’s well worth your time to explore your proposed theory of change, your indicators, and whatever else you can before they’re set in stone. Acting with intentionality at the outset of an evaluation pays big dividends when it’s time to present findings.

5. Always mix your methods.

One thing that I already knew as a researcher, but that was reinforced powerfully at the AEA summer institute, is that it’s always best to use both qualitative and quantitative methods of measurement. Listening to speaker after speaker, it became increasingly apparent that quantitative or qualitative measures alone are simply inadequate for telling the whole truth. Quantitative methods are critical for providing the evidence in ‘evidence-based’. That’s not to say qualitative data cannot be considered evidence, but it is by its very nature more subjective, and requires ‘quantifying’ to become less open to interpretation. Quantitative data will satisfy the number crunchers in the room as well as make your findings look and feel more robust. But it isn’t quite enough. Qualitative data brings the ‘human’ element to the human sciences in ways that quantitative data cannot, because it enables researchers to (often literally) take the perspective of the participant. That makes for insightful narratives and stories that not only bring the results of an evaluation to life, but also give a human voice and face to the ‘hard’ data that’s been brought to the table. At the AEA summer institute, all the best presentations of evaluation results discussed both quantitative and qualitative elements.

How to Kill Your Community Coalitions and Collaboratives: Meet and Talk Meetings

Because CES works with a lot of community collaboratives and coalitions, I attend a lot of community meetings. So, I see the good, the bad, and just plain ineffective. What I have observed is that there are some sure-fire things you should NOT do if you want to fire up your community for change. (Note: for the purpose of this blog, I use the terms coalitions and collaboratives interchangeably).

Meet and Talk meetings are probably the most common meeting killer that I see. In this type of meeting, attendees go around the room and share updates of their organizations, one after another. There is no work done, no direction, and participants don’t really connect in any meaningful way. What participants share in the meeting really could be done through other means like email updates, newsletters, or social media posts. There isn’t really a purpose that brings participants together and if you use this type of structure consistently, you shouldn’t be surprised if the energy, and eventually your attendance, decreases.

Another guaranteed meeting killer is to have guest speakers present at each and every meeting. Although your participants may learn something valuable, this type of meeting has the same effect as the Meet and Talk type of meeting. Participants don’t really connect in any meaningful way, and there is no work done.

After all, isn’t collaboration is THE whole point of a community coalition or collaborative?

Recently, Gabrielle Hawkins-Stewart, a Prevention Support Specialist for Georgia Family Connection Partnership, led a discussion for a group of community collaborative coordinators on how to have effective meetings. This list will highlight some of these ideas along with some of my own suggestions about ways to infuse energy in community collaborative meetings. If you have been having Meet and Talk meetings for a while, you may have noticed that your attendance is down. If that is the case, you definitely want to act quickly and shake things up.

1. Get energy started before people enter the room. Have music playing. Have a bright color table cloth at the sign in table. Place candy and table toys around the room. Start with a quick, fun ice breaker. Introduce new members and help everyone connect, but don’t waste valuable time having everyone introduce themselves.
2. Remember – the whole point of gathering your community leaders and members together is to collaborate. So, make the meeting about collaboration! If you have time, make part of the meeting a work session, perhaps letting committees and workgroups meet briefly within the meeting to have a work session. At the very least, committee and workgroups should provide updates tied to your work plan. This of course, requires you to assign responsibility for your action plan to the committees/workgroups in the first place! If your time is limited, you might want to have longer work sessions every other meeting or perhaps quarterly.
3. If you are going to have guest speakers, require them to attend a certain number of meetings before they get the floor. This will help you limit people who want to get their message out but are not tied to the work of the community collaborative/coalition. If you do have guest speakers, require them to complete an information form that structures their talk so that you can ensure that what they share ties back to the work of your coalition. Then as the leader, when the speaker is done, make sure YOU tie it back to the work. If you do have speakers, you might announce that there is a speaker and tease the topic, but don’t announce the name of the speaker.
4. Every once in a while, it’s a good idea that you as the coalition or the collaborative leader lead the meeting. Use this opportunity to remind everyone why they are there. Share the purpose of the collaborative, share the goals and objectives you are trying to accomplish, the strategies you are implementing to accomplish those goals, and the data you are using to track your progress. Make sure you assign responsibility for the work to work groups and committees and use this time to show them why their work is important to achieving the collaboratives goals.
5. Round table discussions are another way to involve collaborative members in the work. Identify some good discussion questions and group members discuss and report back. Concentrate on questions that are related to the work: the strategies and changes that the collaborative and coalition are putting into place.
6. In order to provide the communication that your participants desire and expect, use different ways to facilitate communication other than taking valuable collaborative meeting time for report outs. For example, once a year, hold a resource fair so that members can share with each other what they do. You might consider limiting the first hour of the resource fair to members, then opening the resource fair to community members.
7. Instead of members sharing details of their upcoming events during the meeting, create a newsletter with a Community Event corner.
8. Encourage communication by sending out an email to those who missed the meeting with the message, “Missed the meeting? Call me (the collaborative leader) or a friend and find out what you missed!”
9. Focus on the community outcomes your collaborative is trying to address (e.g. poverty, graduation rates, foster care, unemployment). Looking at your data only once a year when you update your work plan or write your year-end report isn’t effective. Use your indicator data often, perhaps quarterly, to inform your members about local conditions and help motivate them to stay/get engaged.
10. You might consider asking groups to pay a small member fee to become a member of the coalition; they will have a little skin in the game and perhaps be more likely to attend collaborative meetings.
11. Find resources to help you improve your meetings. Look for professionals in your community who are good facilitators and ask them to facilitate a meeting. SCORE is a nonprofit association and a resource partner of the Small Business Association (SBA) whose members are retired professionals dedicated to helping small businesses through education and mentorship. They have a wide variety of skills that may help your collaborative and your members.

We’d love to hear from you! What tips have you used for effective meetings?