Evaluation Consulting Round-Up

Part 3

“New Directions for Evaluation” is one of the American Evaluation Association’s (AEA) academic journals, and they publish articles on cutting-edge issues in the evaluation world. Recently, they released a special issue focused on being an independent evaluation consultant. This is a round-up of what we learned.

Check out the other articles in this series:
Part 1 – non-evaluation skills you need to be a good evaluation consultant.
Part 2 – managing your consulting business.
Part 3 – maximizing productivity with business processes.
Part 4 – branding and marketing.

Part 3 in this series is about business processes: a standard set of tasks you set up to produce something important for your business. This information comes from Michelle Burd’s article “Build and Improve Business Processes for Evaluation Consulting.”

A business process is a set of tasks, including the staff and materials required, that is used again and again to accomplish a goal. Processes are important for businesses because they help you operate smoothly and effectively. Examples of processes include marketing to clients and tracking staff time.

Michelle Burd interviewed several evaluation consultants at small firms to learn more about business processes in the evaluation world. As you can probably guess from the name of the article, the findings focus on two main areas: building processes and improving processes.

Build Processes

Track

Technology can help small businesses streamline tasks such as invoicing or making payments, and organize important information such as budgets or time tracking. However, using tech doesn’t make a business successful – you need to judge what tool is right for you, and make sure it doesn’t end up taking more time than it’s worth. These programs were used by the consultants interviewed:

• Time tracking: Toggl, QuickBooks, Google Calendar, gtimereport, and Excel

• Accounting: Xero, Quickbooks, and Excel

• Project management: the article only mentioned Excel, but other options include Asana, Harvest Forecast, and Celoxis

Automate

Even though every evaluation is unique, you will find some tasks and processes that are often repeated. These tasks are great candidates for automation: save yourself time by creating reusable templates. Insert boilerplate language into these templates, and then customize them as necessary for each project. Some document templates you might find useful are:

• Scope of Work (SOW)

• CVs and resumes

• Reference lists

• Staff bios

• Ethics applications (such as for an Institutional Review Board)

Automating processes doesn’t have to be complicated. It’s all about reducing unnecessary repetition and duplication of work – don’t reinvent the wheel!

Organize

This is about making your business run like a well-oiled machine through organization and standardization. Four tips were identified:

1. Organize roles among team members to identify who is responsible for doing what. For example, an office manager might prepare monthly budget reports, schedule meetings, and draft proposals.

2. Create file/ folder storage conventions. This includes how to track versions of the same document, how to name files, and which folders to include in a new project. Standard folders within every project could include Data, Billing, Planning, Reports, Research, etc.

3. Leverage technology to collaborate remotely. Some commonly used software to help collaboration are Trello, Slack, Dropbox, Google Suite, Skype, and Zoom.

4. Communicate internally and externally. One evaluation consultant interviewed in the article regularly sends a joint email to the clients and subcontractors working on a project. The email updates everyone on work that’s been completed recently, and what’s planned for the upcoming week, keeping everyone in the loop.

Improve Processes

Business processes won’t stay the same forever – they will need to be tweaked and improved as conditions in the business change. Two ways to improve processes are through evaluation and reflection.

Evaluate

Fortunately, if you are an evaluation consultant, you already know how to evaluate things (it’s kind of your job). Process mapping is a tool to formally evaluate and improve business processes. The steps for process mapping are:

1. Choose the process you will evaluate

2. High-level overview: come up with 4-7 major steps from beginning to end

3. Go into detail: list all the tasks involved in each of the major steps

4. Review the finished map and identify where things are running smoothly, and where things need some work to improve

5. Use real data to help drive improvement (like a good evaluator)

This process map shows the steps involved in sending a monthly update to your email list. The major steps run across the top, and specific tasks are listed under each.

Reflect

When running a business, it’s important to take the time to reflect on your business processes in a formal way so you can be thoughtful and strategic about moving forward. As an evaluation consultant, you might take a day or two once a year to ask yourself these types of questions:

Where have I been?

• What type of work have I done? Did I do the kind of work I really wanted to?

• What’s the market been like? What new areas of work are emerging?

• How have I spent money? How profitable has it been? Did investments bring the returns I expected?

• How effective was my marketing?

• What areas of interest are clients talking about?

• What are my relationships with clients like?

Where do I want to go?

• What new skills will I need?

• What kinds of professional development do I want? Conferences? Writing?

• What kinds of projects do I want to find in the future?

• What do I want to do more of, and less of?

When you’re done, write up a plan for the coming year, and revisit it in 6 months to check your progress and make updates.

In the next round-up, we delve into the world of branding and marketing.

Source:

Burd, Michelle B. “Build and Improve Business Processes for Evaluation Consulting.” New Directions for Evaluation 2019, no. 164 (2019): 69–80. https://doi.org/10.1002/ev.20385.

Email Address
Sign Up

We respect your privacy.

Thank you!

Transforming Evaluation

Part 2

The holidays are over (boo!) and we have all accumulated a few extra pounds, well-intentioned resolutions and a few new gadgets we’re testing out. Maybe some of you are testing out your new virtual reality (VR) headsets? If not, keep reading because this blog post explores augmented and virtual reality and how evaluators can use this technology. If you don’t know what VR is then keep reading – you may find some ideas for transforming your evaluation practice.

What are they and how do they differ? 

Augmented Reality (AR) is just that – elements added to the real world to enhance the user’s experience. There are lots of companies embracing this technology:

• Not sure if a piece of furniture is going to fit in your space? Ikea has you covered with its Ikea Place app.

• Not sure if that wall colour will look right? Benjamin Moore’s Color Capture app lets you visualize the colour on your wall.

The list goes on and on. The point being that AR uses the existing environment and adds information to make a new artificial environment.

VR, on the other hand, does not center on reality but instead generates and transports users to a computer generated, different world. It is a wholly immersive experience where you can leave the world you’re in and enter the world of your making. Not surprisingly companies are beginning to see the power of VR and how it can be used to not just show people their brand and what they are all about, but experience it. Check out this blog for some great examples of how companies are embracing VR technology:

How and why should evaluators embrace AR and VR?

As evaluators we want people to understand what we have uncovered through our evaluations and feel compelled to act on our findings and recommendations. To do this we need to move beyond simply providing information. Most do not truly learn through passively listening or reading  – we learn by doing. AR and VR change the way people see and feel the world and thus can process information in a more experiential way. With this in mind, there are numerous applications for how to apply this technology in our evaluations:

1

Project Management

Sure we can bring people together via teleconference or video conference but what if we could actually bring our evaluation team together in a virtual meeting space? There are lots of evaluators conducting multi-country evaluations, VR provides an opportunity to bring people from various countries together in virtual meeting spaces. Feeling uninspired by your meeting room? Why not meet on the top of Machu Picchu? That is sure to stir your team’s creative juices!

2

Data collection

Similarly, what about organizing virtual focus groups? We all know the amount of time and money that goes into organizing those, only to have a handful of people show up. What if, instead, people joined virtually? It also makes the whole ethics piece around anonymity a non-starter since the person meeting in the space could be an avatar and could remain anonymous. Organizing a virtual focus group also allows the facilitator to project slides, videos or other concepts on the screen for participants to easily see and comment on.

We work with a lot with clients who are trying to understand and capture the stories of the people they work with and how they are making a difference. In the past we have been hired to interview people to try and understand their stories. There are numerous issues with trying to elicit information from people (especially if they are a vulnerable group):

Who are you? (building rapport) – We all know it is important to build rapport with our interviewees. Very rarely do people spill their guts and get into the nitty gritty of their experiences with someone they just met and don’t trust. But in the real world there often isn’t time for the relationship building touted in our university textbooks – sometimes we need information from people now!

The word(s) are escaping me (verbal communication) – Expressing abstract concepts, memories or complicated ideas is difficult and especially difficult for kids, people with cognitive impairment, ESL or low literacy. When we rely on verbal language to describe experiences, we often times collect superficial information or underdeveloped ideas, leading to less robust data.

I can’t remember (recall bias) – Speaking of poor data, recall bias is a real thing and can lead to incorrect or incomplete data.

What if you set your participants up with a 360-degree camera and asked them to document their experience? We already do this using methods like photovoice, but imagine how much more powerful a 3D experience is? What better way to understand the actual or lived experience than to have your participants capture it in real time, so others can be surrounded in their world? It eliminates the need for people to recall and express their stories to strangers and as a result provides more valid data.

Not only do we end up with better data, but we also end up with data that people understand and are compelled to use. Research suggests we retain more information and can better apply what we have learned after participating in VR exercises. Interestingly, emotional understanding and empathy are also found to improve with the usage of AR and VR. Stanford researchers found that people who underwent a virtual reality experience, called “Becoming Homeless,” were more empathetic toward the homeless and more compelled to sign a petition in support of affordable housing than other study participants.

3

Reporting and Use

This blog series is all about how we transform how clients utilize information. If we want people to act on our evaluation findings then let’s compel them by immersing them in the data.

“Data parties” are used to create a more participatory environment for uncovering and discussing findings. This is great if you can get everyone in one spot. If that’s not possible, an alternative might be to use VR to create a space to get people together to discover findings. You could even make a game of it. Imagine a virtual Amazing Race where participants could visit the Great Barrier Reef, the Grand Bazaar in Istanbul, and the pubs in Ireland to discover, discuss, and answer questions about findings or come up with recommendations. VR could also be used to show people alternate realities based on your recommendations; you could walk people through different options or recommendations and what those would look like.

“Whoa, whoa, whoa” you say, “that sounds complicated” or “that’ll be way too expensive” or “that won’t work because <insert naysayer response>.” Just like any technology it starts off complicated until it is simplified. It starts off expensive and then isn’t. The technology is already here. There are 360-degree cameras now that can attach to your phone for less than $300, free software to upload your videos and create your own 3D experiences, and super inexpensive viewers that your audience can use to watch/participate in your experience. Check out Google’s AR and VR site for various products.

AR and VR have the power to transform evaluations from a process that provides information to one that creates knowledge. Just like anything if you don’t embrace it you will be left behind. Change happens; embrace it, leverage it. Will your evaluation practice be Blockbuster? Or will it be Netflix?

Check out Part 1 in this series: Visual Storytelling Through Videos

Email Address
Sign Up

We respect your privacy.

Thank you!

Evaluation Consulting Round-Up

Part 2

“New Directions for Evaluation” is one of the American Evaluation Association’s (AEA) academic journals, and they publish articles on cutting-edge issues in the evaluation world. Recently, they released a special issue focused on being an independent evaluation consultant. This is a round-up of what we learned.

Check out the other articles in this series:
Part 1 – non-evaluation skills you need to be a good evaluation consultant.
Part 2 – managing your consulting business.
Part 3 – maximizing productivity with business processes.
Part 4 – branding and marketing.

Part 2 in this series is about what evaluation consultants can learn from business management experts. This information comes from Matthew Feldmann’s article “Standing on the Shoulders of Giants: Applying Business Management Approaches to Independent Evaluation Consulting.”

A lot of people have done a lot of thinking and researching around how to run a successful business, so thankfully you don’t need to reinvent the wheel. There’s a lot independent evaluation consultants can learn from that body of knowledge. Three big takeaways from this article were: go from good to great, cross the chasm, and never eat alone. Let’s explore each of these.

Go From Good to Great

There are some recommendations that can help you take your consulting business to the next level. Experts have found these 6 practices are the difference between good businesses and great businesses:

1. Level 5 Leadership

You need a leader who is humble, extremely motivated, and dedicated to the organization.

2. First who, then what

Focus on finding staff who are a good fit for the organization – “get the right people on the bus and in the right seats.”

3. Confront the brutal facts

Try to take a clear, unbiased view of the workplace (even though its hard). Focus on doing what you do well and stopping what isn’t productive.

4. The hedgehog concept

This comes from the Greek fable “The Hedgehog and the Fox.” The moral of this story is that a fox knows many things, but a hedgehog knows one important thing. In business, the hedgehog concept means finding your unique niche, then having a laser-focus on this one big idea. Your hedgehog concept is something that you are passionate about, you are the best at, and can make you money.

5. Culture of discipline

Strive for an almost fanatical commitment to the business approach and achieving your goals.

6. Technology accelerators

Capitalize on technologies that can help expand your opportunities, such as social networking or online sales.

Crossing the Chasm

When a new technology comes on the scene, there’s usually a small group of people who jump on board (the early adopters), and a much bigger group of people who wait before buying-in to the new craze (the mainstream market). Early adopters might purchase a new technology before all the bugs are worked out (think of early mobile phones – big, bulky, and inconvenient). The mainstream market, on the other hand, might not know about this new tech yet, or they want to wait until it’s more convenient and cheap (cell phones get smaller, more affordable, and more widely used). The gap between the small group of early adopters and the big mainstream group is called the chasm.

As an evaluation consultant, you are trying to spread a message about your services to a population that might be largely unaware of the value of evaluation. There will be some early adopters of your message, but to truly excel you need to break into the large mainstream market. Your job is to figure out how to cross the chasm.

Never eat alone

Don’t worry my fellow introverts, you’re allowed to eat alone – this just refers to the idea that you need to be taking every opportunity you can to network and build connections with people. Networking is based on generosity toward others by offering them value. Ideally, these relationships are mutually beneficial because you may eventually get more projects and clients as a result of this networking. But don’t keep score! Networking is not a perfectly equal give and take. If you expect a 1:1 return on your generosity, you are setting yourself up to be disappointed. Instead, keep your focus on becoming indispensable to the people around you. Use your unique skillset, contacts, and knowledge to make yourself irreplaceable. Think of the world as a complex web of relationships – you’re trying to make yourself an important strand of that web by being the go-to person for evaluation within your niche.

In my next round-up, we get our feet wet and turn these management concepts into real-world actions with “business processes.” Now get out there and find your hedgehog!

Source:

Feldmann, Matthew L. “Standing on the Shoulders of Giants: Applying Business Management Approaches to Independent Evaluation Consulting.” New Directions for Evaluation 2019, no. 164 (2019): 55–67. https://doi.org/10.1002/ev.20383.

Email Address
Sign Up

We respect your privacy.

Thank you!