drbethsnow

The next domain of competence is technical practice.

2. Technical Practice competencies focus on the strategic, methodological, and interpretive decisions required to conduct an evaluation.

And the first competency in this domain is:

2.1 Clarifies the purpose and scope of the evaluation.

Let’s begin at the beginning. That may seem trite, but I think it’s such a common saying because so often, people want to start somewhere other than the beginning. I cannot tell you how many times I’ve been consulted about an evaluation and the person seeking my advice starts with something like:

• I have a set of indicators and I need to do an evaluation using them.
• I want to do an evaluation of my program but I can’t figure out how to make it into a randomized controlled trial (RCT) because the program is already run.
• I need your help to create a survey to evaluate my program.
• I need to do a developmental evaluation [or whatever the latest trend in evaluation is at the time] of my program.

These are all examples of not beginning at the beginning. Many people seem to think that an evaluation requires a specific method (e.g., a survey) or a specific design (e.g., an RCT). Or they think that whatever the latest trend in evaluation must be the best approach, because it’s new. Or they have data already and they want to use it ¹I just noticed that I‘ve written about this before, more than 4 years ago! Past Beth would be sad to hear that I’m still experiencing this!. But where an evaluation needs to start is with its purpose. Why, exactly, do you want an evaluation? What will you use the findings of the evaluation for? These are the types of questions that I will ask (usually preceded by me saying “Let’s back up a second!”). Because the purpose of the evaluation will guide the choice of approach, design, and methods. For example, if you are interested in an evaluation that will help you to determine to what extent you’ve achieved your goals, and none of your current indicators relate to your goals, then starting with “I have a set of indicators and I need to do an evaluation using them” is not going to get you where you want to be. Similarly, if you want an evaluation that will help surface unanticipated consequences (and I tend to think that evaluations should usually be on the look out for them), then having a set of pre-defined indicators is not going to be what you need (after all, to create an indicator, you have to have been anticipating that it might be affected by the program!). If the purpose of your evaluation is not a developmental one, then developmental evaluation might not be the best approach for you. So clarifying the purpose (or purposes) of an evaluation is something that I do at the start of every evaluation – and something that I check in on during the evaluation, both to see if what we are doing in the evaluation is helping to meet its purpose and to see if the purpose changes (or new purposes emerge) along the way.

Clarifying the scope of an evaluation is also really important, and something that I struggle with. I’m am an infinitely curious person and I want to know all the things! But there just isn’t enough time and resources to look at every possible thing in any given evaluation, so it’s important to be able to clarify what the scope of any given project is. Like purpose, it’s important to clarify the scope of the evaluation with your client at the start, and to keep tabs on it throughout the evaluation. If you don’t have a clear scope, it’s very easy to fall into the trap of the dreaded “scope creep” – where extra things get added to the project that weren’t initially agreed to and then either the costs go up – or the timeline gets extended. It’s not to say that the scope can’t change during an evaluation, but just that any changes to scope should be done mindfully and in agreement between the client and the evaluator.

Working in a large organization like I do, I also find it useful to understand the scope of other departments that do similar work to evaluation (like quality improvement and performance management). This is helpful in ensuring that we aren’t duplicating efforts of other teams, and also that we aren’t stepping on anyone else’s toes. Also, I’ve had the experience of taking on work that really should have been done by another team (i.e., the dreaded scope creep!) and had we not figured this out by clarifying scope, it would have really impaired our ability to deliver on the work that we needed to deliver on.

My team and I have done some work on clarifying what the scope of evaluation is relative to these other groups and I was about to say “and that’s a topic for another blog posting”, but then I remembered that I’m presenting a webinar (based on a conference presentation I gave last year) on that in a couple of weeks! So here’s my shameless plug: if you want to hear me pontificate on the similarities, differences, and overlaps between evaluation and other related approaches to assessing programs and services, register for my webinar, hosted by the Canadian Evaluation Society’s BC Chapter on Friday, September 13 (that’s right, Friday the 13th!) at 12 pm Pacific Time.

Congratulations, Beth!

[…] Cross posted on my other blog. […]

There have been a few times in my life when I decided to do something and then, as the thing approached I thought “What have I done? This is too big and too scary and too hard and I’m totally not going to be able to handle this!”. Moving across the country to do a PhD. Play in a hockey game that lasts for 10 days. Do an MBA part-time while still working full-time. Accepting my current job. As it turned out, all of these were things that I could handle and are things of which, as it turns out, I’m extremely proud! It’s almost like being scared that I’ve bitten off more than I can chew is a sign that I’m about to do something awesome.

My job prior to my current job was fun and I learned a lot and I met some great people, some of whom I’m still good friends with (Hi Heather!). But after 5 years in that job, I’d hit a pay ceiling, I’d learned all that I could learn, and so I wasn’t feeling challenged any more. And then a co-worker of mine told me about a job posting she’d seen that she thought I might be interested in. It was a job doing the same type of work (evaluation in healthcare), but taking it to the next level. A leadership position where I’d get to run a team of evaluators to conduct an evaluation of a massive, multi-organization, multi-year project that has the chance to change the face of healthcare in the region. I was excited by the possibilities this job entailed, so I applied and I got the job. And a few days after I handed in my resignation at my old job I thought “Oh my god, what have I done? I know how to do my old job really well. But there’s so much I don’t know about this new job – I have to learn a whole new area of healthcare AND I’ll be the boss of people and that’s a whole new ballgame for me. What if I can’t do it?” What I should have realized then was, just like the PhD, just like the Longest Game, and just like my MBA, that fear was a sign of a great challenge and I’d shown over and over again that I can rise to a challenge.

The last five years have been really interesting. I’ve learned a tonne about health informatics, about applying complexity concepts to the evaluation of an ever changing project, about governance, about managing people, about managing data when you have a large group of people creating and using a huge dataset, and that’s not even getting into what I’ve learned in terms of the findings of the evaluation so far!

I’ve had the opportunity to collect data from 13 healthcare facilities and counting, I’ve built my team up from 2 to 11 evaluators (all of whom are pretty fantastic, I must say), and I’ve presented my work across Canada, as well as in the US and Australia.

And even after five years, I’m not bored. I honestly feel like we are just getting things rolling and we are improving our processes at every step, and I’m learning so much from all the amazing people on my team, and we are producing information that is actually getting used by decision makers. And there’s so much more still to come.

This is not to say that it’s been easy, or that I will be easy going forward. In a recent presentation I gave about the project at the Canadian Evaluation Society conference, I used this image to represent my experience:

I also often reference that MC Escher painting where the stairs are going up but also going down at the same time as representing what it’s like to work on the project I’m working on. (I can’t put the image here on the blog because I don’t have copyright permission, but here’s a link to the Wikipedia page on it where you can see the image)

But honestly, it’s kind of OK with me. The real world is messy and things don’t always work out how you planned them, but you learn a lot by going along for the ride.

Image sources:

• Fantasy sea monster posted on Pixabay by Barroa_Artworkswith a royalty-free license..
• Number 5 posted on Flickr by Steve Bowbrick with a Creative Commons licnese.

Cross posted on my other blog.

Applauding your engagement in professional networks and the contributions you have made to the evaluation profession and the community of practice, Beth. You really exemplify this competency. Keep up the good work!