[…] Factors that promote use: A conceptual framework […]
allblogs
Dissertation RQ1: To what extent are interpersonal and research factors related to use?
This blog post is a modified segment of my dissertation, done under the supervision of Dr. Tiffany Berry at Claremont Graduate University. You can read the full dissertation on the Open Science Framework here. The rest of the blog posts in this series on my dissertation are linked below:
- Factors that promote use: A conceptual framework
- Defining evidence use
- Overview of my dissertation study: sample, recruitment, & measures
- Question 1: To what extent are interpersonal and research factors related to use?
- Question 2: To what extent do interpersonal factors relate to use beyond research factors?
- Question 3: How do researchers and evaluators differ in use, interpersonal factors, and research factors?
To answer this research question, I examined correlations between self-reported interpersonal (i.e., relationships, communication, cooperative interdependence, commitment to use, stakeholder involvement) and research/evaluation factors (i.e., relevance, rigor) with use (i.e., instrumental, conceptual, and process use). These correlations are shown in the figure below.

Overall, all of the interpersonal factors except for two items in the stakeholder involvement scale were moderately correlated with use, with correlations ranging from r = .19 to r = .38. Both research relevance and rigor were correlated with use, albeit relevance (rs between .24-.34) was more strongly correlated with instrumental and conceptual use than was rigor (rs between .15-.22).
These findings support two prior hypotheses I had coming into this research study:
- Interpersonal factors are perhaps more important for process use than for instrumental and conceptual use.
- Relevance is more strongly correlated with use than rigor, although this was only found for instrumental and conceptual use.
I also examined some demographic factors to see how they related to use. Partnerships that have been together longer had greater instrumental (r = .20), conceptual (r = .20), and process (r = .25) use, and also reported greater rigor (r = .16) and less conflict among partners (r = -.20). Partnerships with more members reported greater instrumental (r = .14) and conceptual (r = .15) use, lower quality relationships (r = -.18), lower commitment to use (r = -.17), and less conflict among partners (r = -.20). Interestingly, those reporting using an RCT reported higher levels of process use (d = .47), instrumental use (d = .39), and conceptual use (d = .45).
Lastly, I asked participants to respond to an open-ended question about the quality of relationships among members. These responses were coded into four levels of relationship quality, which you can see in the table below. Although not statistically significant, there was a slight increase in use across the four levels of relationship quality.

Why your online evaluation capacity building efforts are doomed to fail
This post was written for, and originally appeared, on AEA365.
Hello my fellow evaluators. My name is Chris Lysy, and I am a freelance evaluator, designer, and trainer based in Cary, North Carolina. My consulting work involves helping organizations deal with the virtual shift (shifting programs from the “real world” to the “virtual world”).
Over the last couple of decades lots of organizations have evolved from being fundamentally place-based operations to global digital entities. An evolution that started well before COVID.
Yet for some reason we still have this image of an organization being a place where people who work together regularly come into face to face contact.
So why are your online efforts doomed to fail?
I believe we consistently underestimate how much the social environment has changed as the virtual world has grown.
- A 60 minute webinar is not the same as a 60 minute in-person brown bag.
- A virtual summit is not the same as a multi-day in-person conference.
- A 30 minute zoom chat is not the same as a cup of coffee with a colleague.
It’s similar, not necessarily better or worse, but different.
A lot of what has been written about evaluation capacity building is focused on what concepts to cover. It assumes that evaluators know how to effectively engage people at an organizational level. For the pre-virtual organization, that could mean showing up with free doughnuts in the lobby or hosting a large event in the “big conference room.”
But in order to bring the same level of change in the virtual world, we have to approach the capacity building challenge differently. Or else our efforts will be doomed to fail.
Capacity Building Requires Community Building
Most organizations are not full of vibrant virtual communities. IRL (“In Real Life”) when people get together, community happens. Not so in the digital world. People tend to know their teams (from regular Zoom calls) but they might not know anyone else outside their project bubble.
So if you want to engage a certain community of people (say project leaders or internal data people) you have to start by building that specific virtual community.
A Dead Forum is NOT a Community
Yes we all have lovely dreams of an active forum where people go to ask questions and support one another. But the reality is that good forum communities are hard to build and often require far more people than you even have in your organization.
A Webinar Lecture Series is NOT a Community
Somebody wants a community. So they create a webinar lecture series with topics of interest for that specific community. That might work for community building IRL but it doesn’t work the same online. Do this instead.
- Hold webinar conversations, not lectures.
- Source presenters from inside the prospective community.
- Hold panel webinars to distribute the content responsibility and increase exposure to peers.
- People need to see/hear people to connect, so encourage cameras/headsets (just don’t require them!).
An Email Newsletter is a Good Place to Start
It’s easy to create and just shows up in their email inbox (no need for them to visit a specific site or remember any kind of password/username). You can source newsletter content from the community and it pairs really well with a conversational webinar strategy.
Want more guidance?
I created a short practical guide.
You can download it here: Virtual Community Building 101
Better Storytelling with the Same Data: Upgrade that Board Packet!
Kristen Summers is the Senior Grants Manager at Saint Luke’s Foundation and a Depict Data Studio student. She emailed me an example of how she’s revamped her organizations grants docket and I knew I had to showcase her hard work. Keep up the great work Kristen! – Ann
_________________
I work at a grantmaking organization and it is my job to update the board three times a year on the grant applications we are considering for awards, the grantmaking budget and other information.
You will see below an example of the grants docket (slate of grant proposals under consideration) as well as a grantmaking budget summary. I am almost too embarrassed to show these!
Fortunately, after completing about 70% of the Dashboard Design course, I was able to revitalize a tired spreadsheet into an information powerhouse and a colorful budget visualization into a professional presentation.
It’s the same information, just presented differently!
The Docket
Before: Gridlines Galore and No Visualizations
This docket was printed on 11×17 paper for easier readability and called a “placemat”.
Gridlines, some highlighted cells in yellow to draw the eye.
A lot of information and a bit overwhelming to take in.
Applications were presented alphabetical by organization and not grouped in any way.

After: Fewer Columns, Bigger Impact
When I started the Dashboard Design course with Ann, my intention was to revamp our organizational dashboard and not even this document (once I get that done, I will have to write another blog!).
But the course showed me what little changes I could make to have a big impact.
Although we do not compare numbers over time in this document as it is just a list of our applications, I was still able to use some helpful visualizations, icons, colors, and conditional formatting to tell a story.


A summary of the changes made:
- Added logo and subtitle
- Used brand colors in headings and their corresponding icon
- Minimized the use of gridlines
- Removed highlighting of a column but used font colors for key info
- Grouped applicants together by type of grant (general operating vs. project)
- Hid the column with the organization budget, but added an icon set of pie charts via conditional formatting
- Correct alignment for numbers vs. text
- Deviation bar visualization to show the percent change of the current request from their previous award instead of just expecting the reader to do the math
- Added a budget summary chart at the bottom
This resulted in a much more well-received document with lots of compliments from board members!
I have begun creating a cohesive aesthetic for all the documents I produce for the board to give them the information they need but in the most streamlined way possible.
The Budget
In 2019 I developed the below budget visualization to up my game a bit from a spreadsheet that board members had a hard time understanding.
This was an improvement over the previous version, but Ann’s course challenged me to turn it up a few notches.
Before:

After:

There were not many “big” changes to this in terms of conditional formatting or visualizations, but it has definitely been toned down.
The biggest change was that I replaced the text explanations with icons to illustrate where we are in the budget, which the board members appreciated.
As you can see, there is that cohesive title, subtitle, font, and color choice to keep the branding in line.
Take the Time to Take the Course!
I am pleased with what I am able to do now that I have completed Dashboard Design. The lessons were fun and I was able to put my skills to the test in a matter of weeks!
Now I am enrolled in Simple Spreadsheets which will really provide me a good base as my role stretches me to do more community engagement data evaluation and learning. Thank you, Ann!
Connect with Kristen
LinkedIn: @summerskristen
Lift every Voice 216: https://www.facebook.com/216lift
Saint Luke’s Foundation: https://www.facebook.com/saintlukesfoundationcleveland
Business Tools You Need to Run Your Evaluation Consultancy
Starting a company is daunting. Where do you even begin? When my business partner, Shelby Corley and I started Three Hive Consulting we had A LOT of questions.
We knew (and still believe) that evaluation is a means for driving organizational impact, so starting an evaluation consultancy seemed obvious. What wasn’t as obvious was the HOW. How do we market ourselves? How do we determine our pricing structure? How do we find people to help do the work?
Often it is some of those smaller, more operational type decisions that we’d get stuck on. How do we track our time? How do we track accounts receivable and payable? How do we <insert business process>?
This article will save you the hours of research and provide you with a list of tools that help us run our evaluation consultancy.
1. Communication platform: Microsoft Teams
Microsoft Teams was a game-changer for us. Prior to using Teams, we were sending multiple emails a day to communicate with each other, struggling with versioning control on documents and spending way too much time trying to find where documents were located.
Now that we use Teams, we set up a team site for each evaluation project that organizes all the documentation and communication for that project in one spot. Our communication and workflows for each project are much more streamlined. Now we only use email for external communication.
Plus, more and more of our clients are using Teams and establishing us as guest users on their Teams site, which again means more streamlined external communication with them.
2. Time Tracking: Harvest
Let’s be honest: time tracking is a necessary evil as a consultant. I have to say that Harvest has made it a whole lot less evil. Prior to Harvest we used Excel, which was fine but not ideal.
What I like about Harvest is that I can have a timer on my desktop or even my phone that I can easily open to start tracking. It even lets you know when you may have forgotten your timer and nudges you to adjust it accordingly. The user interface when you log into the actual website is really helpful as well. There is one dashboard that shows us all our projects, our budget, what we’ve spent (billable amount and costs), and how much is left. We can also add contractors as users so all the project costs are tracked in one place.
We also use Harvest to invoice which makes invoicing easier for projects that are billed by the hour. Lastly, Harvest also integrates with our accounting software, which again streamlines processes.
3. Accounting platform: QuickBooks Online (QBO)
Prior to switching to QBO we were living in the accounting dark ages (think printing off receipts and other financial documentation and physically delivering that in an envelope to our accountant).
This antiquated process meant not only were we living in the dark ages in terms of process, but also just in the dark. We did not have timely insights into our finances to help guide our business decisions.
After switching to QBO it was like a light was turned on and we could finally see! Now we have a shared online platform where both business partners and our accountant can login to quickly and accurately assess what money is coming in and what money is going out and we can easily track what that looks like over time. We can even forward emails with receipts to QBO for automatic processing to lessen the bookkeeping workload.
4. Scheduling: Calendly
As evaluators, we are often scheduling meetings or interviews. I’ve eaten up a lot of time in the past emailing back and forth on potential meeting dates and times that would work to meet. Many companies will hire assistants or scheduling coordinators just to take on these tasks.
Calendly has saved us a tremendous amount of time and money by automating the scheduling and booking process. It makes scheduling interviews or focus groups so much easier for participants, too. Calendly connects with your calendar and is an add-in to your email. The email add-in means while you are trying to schedule a meeting you can just click on the Calendly button in your email and it will insert your availability (you also have the option to modify your availability if your calendar shows blocks that are inaccurate).
When you send the email to the recipient, they can then select the time and date that works and it will automatically schedule the meeting in both calendars. If you link to Microsoft Teams it will also insert the link that people need to use to join the meeting. Voila! No more emailing back and forth about who is available when.
5. Password Management: Zoho Vault
Managing passwords is the bane of my existence. Managing passwords in a company with multiple individuals gets even more complicated. In fact, all of the tools I have talked about in this article require passwords!
Thankfully Zoho offers a product called Zoho Vault that has helped make the password management process easier. With Zoho Vault, you sign-in to a “vault” that stores all your passwords for the various products. You can add people in your company as users and then select which ‘secrets’ (aka passwords) each person has access to and their access permission level to that secret.
6. Data Analysis: Excel, R, and ATLAS t.i.
Evaluation means we collect and work with a lot of data – both quantitative and qualitative. We still mainly use Excel for a lot of our analysis. You can run a lot of basic statistical analyses in Excel and also do some pretty incredible data viz in it. The best part with Excel is you don’t have to pay reoccurring fees to use it and everyone pretty much has it so you can also share reports made in Excel. For more complicated stats we use R, which is free!
We also use Excel for some basic qualitative analysis. For any project where we need to manage and analyze larger amounts of qualitative data we use ATLAS.ti. ATLAS.ti has recently introduced a web-based platform that we expect will see continued rollout of the features we’re used to in the desktop version.
7. Project Management: Excel and Microsoft To Do
We are still trying to figure out what project management tools will work best across Three Hive. We have tried a few like Harvest Forecast, but haven’t landed on one that fits quite right.
In the meantime, we use Microsoft To Do, which integrates with Microsoft Office 365 applications (i.e., Microsoft Teams), to plan and assign tasks. I’m told Microsoft Power Automate is also a useful tool that integrates with Office 365 that automates businesses processes and workflows, but we haven’t explored that yet.
8. Virtual Facilitation: Mentimeter, Mural, and Microsoft WhiteboardMural
In addition to being communicators, accountants, administrators, project managers, and analysts, evaluators also need to be good facilitators – my office is full of markers and sticky notes.
When the pandemic happened, I needed to figure out how to facilitate virtual whiteboard sessions. I’ve tried out Mural and really like it, but again once your free trial runs out you are stuck with yet another monthly charge.
Microsoft Whiteboard is a function that integrates with Microsoft Teams. I haven’t used it too much yet, but am keen to learn more. Another tool I have been using while facilitating to help engage with the audience is Mentimeter. If you need to wake people up, try a quick poll using Mentimeter to re-engage them.
If you are just starting your evaluation consultancy and are a team of one you may not need some of these tools; Excel may be all you need. However, if your evaluation consultancy grows, you will find that your needs will too.
A growing evaluation consultancy means more people; more people means more processes and systems; more processes and systems means more tools to support people to do their best work. As Kevin Fishner of HashiCorp put it in this article:
“A company is a collection of both its people and its systems. People can come and go over the years, but the systems they put in place and gradually refine over time become part of the company – and companies ultimately compete based on if those systems are strong or not.”
Make sure to sign up for the Eval Academy newsletter for more evaluation consulting experience shares.
Sign up for our newsletter
We’ll let you know about our new content, and curate the best new evaluation resources from around the web!
We respect your privacy.
Thank you!