allblogs

This article is rated as:

You’ve determined that you need a survey to gather information from a specific population you serve. But where do you start? I recently read the book Designing Quality Survey Questions (2019) by Sheila B. Robinson and Kimberly Firth Leonard, which if you are still a survey design novice, would be a great resource.

In the book, Robinson and Leonard discuss the iterative process of designing a survey. The book is divided into three parts: planning and predesign, drafting questions, and finalizing the survey. Each of these parts is seen as a loose phase in survey design and development; I emphasize loose as this is an iterative process. To best cover the three main components of survey design, I have chosen to follow a similar flow. I found the drafting questions and the finalizing of the survey sections more useful in the drafting of a survey, however, the planning and predesigning section is still an important aspect in survey design and does need to be covered.

Why quality survey design?

The book begins with why survey design is important before the authors even discuss how to design quality surveys. They emphasize the need for quality surveys to get people to want to complete your survey as people are now inundated with survey requests. Part of Robinson and Leonard’s aim in this book is to help you improve on your survey design which in turn would help improve respondent experience and data quality. Throughout the book, they emphasize respondent experience over ease of analysis. The authors see respondent experience as imperative to quality survey design. I see their point here, as frequently I haven’t finished a customer experience survey because my choices didn’t make sense, they had too many textboxes, or the survey itself was too long.

Articulate the purpose

In the planning and predesigning phase, the aim is to understand and articulate the purpose of the survey, along with what the survey can measure, and start to understand the survey respondents. At this phase, the evaluator would determine what knowledge they hope to gain through the use of the survey. The authors included evaluation question(s) that the survey would inform, how the information will be used, and who will use that information as part of the clearly articulated purpose of the survey.

Ensure the survey is the right tool

After articulating the purpose, this is when the evaluator would determine if a survey was the right tool to gather the information. Robinson and Leonard articulate the purpose as 1) understanding why you have chosen to use a survey, and 2) outlining what is planned for the results to help ensure a survey is the correct tool. As they view survey design as iterative, you could reverse these steps if that process made more sense to you. Personally, I think these are really happening at the same time.

The authors then review what can be measured by a survey, including respondent’s attributes, behaviours, abilities, and thoughts. The authors also articulate the advantages and limitations to using the survey tool.

I appreciate the authors spending time outlining the advantages and limitations of the survey tool, to help the evaluator determine if this is indeed the tool for the evaluation questions and potential respondents. I wouldn’t spend time developing a survey when interviews would be a better method.

Survey respondents

Robinson and Leonard devote an entire chapter to understanding respondents, which makes sense to me as the whole purpose of the survey is to gather information from this group, so respondents need to be understood as in-depth as possible by the evaluator. The authors start this chapter with the four cognitive tasks respondents must use in answering surveys: comprehension, retrieval, judgement, and response.

They then examine the potential respondents’ willingness and ability to participate in a survey. They ask questions such as:

• Do the types of questions and the nature of the questions encourage respondent participation?

• Can the potential respondents remember that information, or understand the questions and the language used?

To help with comprehension, the authors do mention the QUAID (Question Understanding Aid) tool that was developed by the Institute for Intelligent Systems at the University of Memphis. This tool can give some feedback on the language used and may be helpful with comprehension. Robinson and Leonard stress that with any tool they suggest, it doesn’t replace field testing or getting feedback from other people in the process of survey design but may help get your tool closer to a final version prior to pre-testing. The authors cover pre-testing the survey later in the book, after drafting survey questions.

The authors then discuss the importance of understanding the context and culture of the potential respondent population. There are several questions the evaluator should consider, such as:

• What is the current political, environmental, economic, organizational, and cultural context?

• How does the cultural background of the evaluator differ from that of the respondents and how can the evaluator be respectful and responsive to this?

• What about the power dynamics; who had the power and privilege or at least who has the perceived power and privilege?

Overall, I found this book a helpful resource in survey design. This book is great for someone new to survey design and could be a great resource for those more experienced as they do give many examples and list other resources for further information at the end of every chapter. I found the later chapters around crafting survey questions and drafting the survey more useful and will review these chapters and any tips and tricks that would be useful in a later article.

Stay tuned for Part 2 of survey design that will explore how to draft quality survey questions.

This article is rated as:

While looking for my next audiobook, I came across Gladys Rowe’s podcast, Indigenous Insights: An Evaluation Podcast and an Indigenous perspective on evaluation piqued my interest. The podcast is available on Spotify, Audible, and on her website gladysrowe.com. Rowe is a member of Fox Lake Cree Nation in Northern Manitoba and holds a PhD in Interdisciplinary Studies from the University of Manitoba. For further information on Rowe check out her bio on her website.

If you are hoping to get a step-by-step process on how to do Indigenous evaluation, this is not the podcast for you. The episodes are more about experiences in doing Indigenous evaluation than a process to follow. It is about what each guest has learned in their journey. It is more akin to a learning circle where you are listening to an elder share their experiences and you are expected to reflect on it and find your own learnings. This aligns with Rowe’s aim for this podcast, to provide a space for those in Indigenous evaluation to learn from one another and to each build our own Indigenous evaluation bundle. I think the learning circle style furthers Rowe’s desire to elevate Indigenous voice and experience as I found this type of learning common when participating in Indigenous teachings.

If you are looking for somewhere to start to get your feet wet in Indigenous evaluation, this might be a good place. The podcasts are easy to follow. I found the storytelling format to be engaging. The sound quality is consistent so the speakers can be heard throughout. This is also not a podcast you would need to listen to most of the episodes in order, so you could jump around to the guests you would find most interesting.

On each episode, she talks with a different researcher or evaluation practitioner about their experiences in Indigenous evaluation. Most of the episodes are approximately 40-50 minutes long, during which she hosts a discussion with one or two guests. The first episode is the shortest of the series as it serves as an introduction to who Rowe is and the aim for the podcast series.

I like how Rowe positions Indigenous evaluation in episode one. She describes Indigenous evaluation as being grounded in Indigenous ways of knowing, being, and doing. “The projects, questions, methods, and meaning making are relational, iterative, and lived deeply within our hearts and spirits.” She sees Indigenous evaluation being more than just bringing methodologies and tools to complete this work but needing to bring her whole self. I found bringing my authentic, whole self into Indigenous work is needed. Through my experiences with Indigenous organizations, I have found that if you didn’t come with your authentic self, you won’t be trusted or ever fully accepted by the community.

Rowe also emphasized the need for evaluation to ensure Indigenous community priorities are central and the “wisdom of those with lived experience” are highlighted. I found that Bremner, her first guest in episode two, further expands on these ideas in episode two and makes some compelling points through his use of stories and examples.

In just the first couple episodes, you can get a feel for what the flow may look like for this series. In the first episode, Rowe talks about her relationship to the land, connection to the Indigenous community, her educational and work background, and her learnings in conducting Indigenous evaluation. A similar flow continued in the next episode with Larry Bremner. This is like providing a personal land acknowledgement which includes the individual’s relation to the land, the historical people who lived on the land, and who their own ancestors are. If you are interested in developing your own land acknowledgement you could use some of the format found in the introductions of these podcasts or the University of Saskatchewan has their own video series of how to create one.

In episode two, Larry Bremner, through his use of stories, discussed how evaluation needs to come from the community and benefit that community. He described Indigenous evaluation as being about “social, environmental, and economic justice.” For evaluation to benefit the Indigenous community, he saw the need for the evaluation and its priorities to be set by the community. Frequently evaluation was done by external bodies and did not reflect the priorities or realities of the community. Therefore, many of these evaluations had no lasting effect on the community. The benefits for the community need to be first and foremost in the evaluation. I found it interesting that he then linked the use of evaluation to further colonize Indigenous peoples which does help explain why there is a lack of trust in evaluation processes.

Part of Bremner’s evaluation experience related to the importance of the evaluator as an embedded member of the community, in this way, the evaluator becomes part of the story. The evaluator is an active participant of the community and is a trusted member of the community. To me, this relates back to Rowe’s comments about bringing your whole self into the work. I have seen how Indigenous people will not trust and accept someone into their community who is seen as fake. A program can be successful, or not, based on who the facilitator is and what they bring of themselves to the project. If the evaluator isn’t genuine, it will be seen, and community members will not engage with that person.

By the evaluator being part of the community, the evaluator knows which ceremonies it is appropriate to participate in and when not to. Through this participation and observation, key findings can be witnessed that would never show up through a questionnaire. Bremner did give an example of a ceremony he didn’t participate in but could observe the outcome of that ceremony through the conversation with the participant later. Again, the participant only shared this knowledge because he was a trusted part of the community.

Bremner provides some insight into the concept of knowledge ownership, where the owner is credited for that knowledge. He saw the other side to that being where knowledge is taken and used elsewhere without giving credit to those who originally gave it. I found it interesting that he connected this to the appropriation of Indigenous knowledge. Ultimately, I believe this is frequently the concern with evaluations that happen in Indigenous communities.

There are other learnings you could glean from Bremner’s stories. These are just a few of the topics he discussed.

Overall, I found these episodes to be packed with ideas and concepts that could further influence how anyone might approach evaluation or really any work within Indigenous communities.

Episode 1: S01E01: Indigenous Insights – Making Introductions

Episode 2: S01E02: Indigenous Insights – Larry Bremner

Have an evaluation podcast you love? Drop it in the comments below!