Factors that promote use: A conceptual framework

This blog post is a modified segment of my dissertation, done under the supervision of Dr. Tiffany Berry at Claremont Graduate University. You can read the full dissertation on the Open Science Framework here. The rest of the blog posts in this series on my dissertation are linked below:

1. Factors that promote use: A conceptual framework
2. Defining evidence use
3. Overview of my dissertation study: sample, recruitment, & measures
4. Question 1: To what extent are interpersonal and research factors related to use? 
5. Question 2: To what extent do interpersonal factors relate to use beyond research factors?
6. Question 3: How do researchers and evaluators differ in use, interpersonal factors, and research factors? 

There has been a lot of research examining factors that promote evidence use in both research and evaluation. Multiple frameworks for categorizing the different factors that promote evidence use have been proposed in the past (Alkin & King, 2017; Nutley, Walter, & Davies, 2007; Palinkas et al., 2011). In particular, King and Alkin (2018) stress that use occurs when certain types of users interact with certain types of researchers, who conduct research in a certain way. These categorization schemes can be summarized into intrapersonal (e.g., researcher and practitioner factors), interpersonal, organization, and research factors, which I’ve hypothesized their relationship in the figure below.

Note: The paths from interpersonal and research factors to evidence use were the only ones I tested in my dissertation study.

In this conceptual framework, the researcher and practitioner factors lead to evidence, but they also lead to evidence through research factors (e.g., rigor, relevance, timeliness, credibility) and organization factors (e.g., organizational capacity and size), respectively. Furthermore, researchers and practitioners must also work together to promote evidence use through interpersonal factors (e.g., relationship quality, communication, commitment to use, stakeholder involvement, cooperative interdependence).

Below, I will briefly discuss some of the literature on these five factors and how they promote evidence use.

1. Organizational factors: Various researchers and evaluators have suggested organizational context affects evaluation use, including the age, development, and size or scope of the organization or program; institutional arrangements and autonomy within the organization; and community influences on the program (Alkin & King, 2017).

2. Research/evaluation factors: Much research has examined research rigor on promoting use, but ample evidence suggests that (a) researchers and practitioners differ on their definition of rigor and (b) rigor is perhaps necessary but not sufficient to promote use. Rather, the research or evaluation must also be relevant, trustworthy, useful, and accessible to promote use (Alkin & King, 2017; Tseng & Gamoran, 2017a).

3. Practitioner factors: These factors include the practitioners’ predispositions about research or evaluation, including their prior experience with evaluation (Taut & Brauns, 2003) and anxiety towards evaluation (Donaldson, Gooler, & Scriven, 2002), as well as their interpersonal skills and the personal factor. The personal factor is “the presence of an identifiable individual or group of people who personally care about the evaluation and the findings it generates” (Patton, 2008, p. 66). When such stakeholders are present, evaluations are more likely to be used. However, little research has empirically examined the personal factor’s effect on evaluations (Fleischer, 2014).

4. Researcher/evaluator factors: Similar factors for the practitioner are also necessary to see in the researcher or evaluator, including their prior experience working with practitioners and their interpersonal skills. Like the personal factor, the interpersonal factor—the presence of an evaluator who personally cares about promoting use—has received some attention in the literature on promoting use.

5. Interpersonal factors: There are a variety of interpersonal factors shown to promote use.
   • Communication quality: First, partners need to communicate effectively, which involves clear, frequent, and wide discussions across a multitude of media (Fleischer & Christie, 2009; Henrick et al., 2017; Maloney, 2017; Nutley et al., 2007). Furthermore, researchers need to communicate recommendations or implications of the evidence produced to support practitioners’ decision-making (Cousins & Leithwood, 1986; Fleischer & Christie, 2009; Maloney, 2017; Masaki, Custer, Eskenazi, Stern, & Latourell, 2017; Nelson et al., 2009).
   • Practitioner involvement: Evaluators have increasingly recognized the importance of stakeholder involvement to promote use (Alkin & King, 2017; Froncek & Rohmann, 2019; K. Johnson et al., 2009). However, there are a wide variety of theories and approaches to stakeholder involvement that differ in their approach, and research has yet to fully examine whether and how practitioner involvement promotes evidence use.
   • Relationship quality: Ample research on collaborations have stressed the importance of relationships, identifying it as one of the core dimensions of an effective partnership and factor to promote use. Research on research-practice partnerships have thus far supported the importance of relationships in promoting use. However, much of this research treats relationships simply as trust, mutualism, and long-term commitment, which are important but insufficient for true relationships. Relationships also involve positive emotions such as liking one another, feeling close to each other, and feeling respect, dependability, warmth, and overall friendship (Berscheid, Snyder, & Omoto, 2004; Carmeli, Brueller, & Dutton, 2009; Dave et al., 2018; Marek, Brock, & Savla, 2015; Stephens, Heaphy, & Dutton, 2012). This allows partnerships to go beyond transactional relationships to communal relationships. Ample research from developmental, organizational, and social psychology support the importance of relationships in partnership work.

Although my dissertation only tested a few of these pathways in my study, my hope is this conceptual framework can help others examine other research questions in relation to evidence use. For example, to what extent do research factors partially mediate the relationship between researcher factors and evidence use? As another example, to what extent do both practitioner and researcher factors lead to interpersonal factors to promote use? These are important questions that will help the field better understand how to promote evidence use in research and evaluation studies.