Survey Design and SoTL
Things to consider before drafting a survey
- What are some general concepts I am interested in? (e.g., how useful were parts of the course? How much do students feel they learned from the course as a whole? What suggestions do students have about improving the course?)
- What variables can I use to measure these concepts?
- What are the specific questions which help me get at these concepts (e.g., how useful were the lectures? How useful was assignment 1?)
- What demographic questions can I ask that help me address my research questions (e.g., gpa, expected grade, gender, major)?
- What quantitative or qualitative analyses can I use to answer the questions that I have?
Survey Question Types
Open ended (i.e., fill in the blank)
- Short answer
- Multiple choice (e.g., indicate your major) – Easy to gather data for variables when only one answer is possible
- Checklist (e.g., select all pre-requisite courses you have taken) – Easy to gather large amounts of data especially for multiple grouping variables when each answer can be present/absent
- Rating scale – easy to count for small scales and easy to convert to numbers for quantitative analysis
- e.g., 1=strongly disagree; 2=disagree; 3=neither agree/disagree; 4=agree; 5=strongly agree
- Keep category wordings as “evenly spaced”, balanced, and non-overlapping as possible
- If absolutes (e.g., Always, Never) are used are these likely or meaningful?
- 4-7 choices are most typical
- Choices should generally be qualitative or both numerical and qualitative (e.g., 1=strongly disagree, 2=disagree… etc.)
- Is a neutral response choice (odd number of categories) desired or is “forced choice” preferred (even number of categories)
- Consider if a do not know/not applicable option should be included
Best Practices in Creating Questions
- Relate to variables/concepts you are interested in and analyses you plan to use
- Are clear and easy to interpret by intended reader. Have definitions and/or clarifications if required
- Answers can be recalled easily by participants (e.g., didn’t happen too long ago, isn’t too hard to estimate)
- Word questions positively (e.g., how useful was the assignment?) rather than negatively (e.g., how useless was the assignment?)
- Questions should be concise
- Not leading or biasing (do not “suggest” an answer to the question, word as neutrally as possible)
- Not “double barrelled” (e.g., “do you like science and math?”)
- When reviewing your questions, ask the following:
- Have all concepts of interest been represented by different questions?
- Are all general concepts sufficiently covered by multiple questions (at least 3-5) to assess them?
- Can the literature help inform question wording and/or variable choice(s)?
- Does each question have a clear purpose?
- Do I have an idea of how this question will be used in the analysis and why I’m asking it?
Using Validated Instruments
- Even if it is “validated” not all validated scales are created equal and modification of an instrument will require further work and reassessment
- Questions to consider regarding the validity of an instrument:
- Where was it published? Has it been improved over time?
- How frequently is it used/cited?
- What methods were used to validate the instrument? How rigorous was this and how large/generalizable was the population used?
- How well-suited is this instrument for your study?
- How closely does the construct in the instrument match your interest?
- Has it been tested on a similar population to mine?
- What adaptations might be needed for my own research?
(Note: changing an instrument will “break” its validation)
1=Disagree, 2=Somewhat Disagree, 3=Neither Agree Nor Disagree, 4=Somewhat Agree, 5=Agree, Not Applicable or Do Not Know
1=Almost Never, 2=Seldom, 3=Sometimes, 4=Often, 5=Very Often, Not Applicable or Do Not Know
1=Very Poor, 2=Poor, 3=Satisfactory, 4=Good, 5=Very Good, Not Applicable or Do Not Know
1=Very Little, 2=A Little, 3=Somewhat, 4=A Lot, 5=Quite a Lot, Not Applicable or Do Not Know
1=Mostly Not Useful, 2=Not Very Useful, 3=Somewhat Useful, 4=Very Useful, 5=Extremely Useful, Not Applicable or Do Not Know
For more in-depth information on scale development and survey design, we recommend the following resources:
DeVellis, R. F. (2003). Scale Development: Theory and Applications (Vol. 26). Thousand Oaks, CA: Sage Publications.
Krosnick, J. A., and Presser, S. (2010) Question and Questionnaire Design. In P.V. Marsden, and J.D. Wright (Ed.), Handbook of Survey Research (2nd ed, pp. 263-313). Bingley, WA, UK: Emerald Group Publishing Limited.