Pilot Tests and Focus Groups
Similar to pretests, pilot tests are used to evaluate how a sample of people from the survey population respond to the questionnaire. For a pilot test, surveyors typically contact a large number of people so that potential differences within and across groups in the population can be analyzed. In addition, pilot tests for many surveys test the full implementation procedures (e.g., contact letters, incentives, callbacks, etc.). Pilot tests are usually conducted well in advance of when the survey will be fielded so that more substantial changes to the questionnaire or procedures can be made. Pilot tests are particularly helpful when surveyors are testing new questions or making substantial changes to a questionnaire, testing new procedures or different ways of implementing the survey and for large-scale surveys, such as the U.S. Census.
Focus groups are very different from pilot tests because people discuss the survey topic or respond to specific questions in a group setting, often face to face (though online focus groups are sometimes used). When conducting focus groups, the surveyor typically gathers a group of people and asks them questions, both as a group and individually. Focus group moderators may ask specific survey questions, but often focus group questions are less specific and allow participants to provide longer answers and discuss a topic with others. Focus groups can be particularly helpful in gathering information before developing a survey questionnaire to see what topics are salient to members of the population, how people understand a topic area and how people interpret questions (in particular, how framing a topic or question in different ways might affect responses). For these types of focus groups, the moderator typically asks broad questions to help elicit unedited reactions from the group members, and then may ask more specific follow-up questions.
For some projects, focus groups may be used in combination with a survey questionnaire to provide an opportunity for people to discuss topics in more detail or depth than is possible in the interview. An important aspect of focus groups is the interaction among participants. While focus groups can be a valuable component of the research process, providing a qualitative understanding of the topics that are quantified in survey research, the results of focus groups must be interpreted with caution. Because people respond in a group setting their answers can be influenced by the opinions expressed by others in the group, and because the total number of participants is often small (and not a randomly selected subset of the population), the results from focus groups should not be used to generalize to a broader population.