Survey Terminology

Below is a list of common survey terms and how they are used in our office. Please note that terminology has been simplified and may vary from other sources.

  • Target Population
    The group of people you wish to study with your survey. Example of the target populations would be all undergraduate students at Clemson or all faculty in the College of Business.
  • Sampling Frame
    The list of people from which your sample will be drawn. A sample frame largely overlaps with the target population. By default, our office excludes the following students from the student sample frame: 1) students under the age of 18 and 2) students who have asked for their directory information to remain confidential.
  • Sample
    The term sample is used in two ways, which can cause confusion.
  • During the survey design and data collection phases, the sample of potential respondents is the list of people who are invited to take the survey. Samples are randomly drawn from the sample frame and designed to be representative of the desired population.
  • Once the survey data is being analyzed, sample more commonly refers to the collection of valid respondents and their responses (also called observations). When making statistical inferences, be sure to use the characteristics (such as the size and representativeness) of the actual respondents. See sample size below.
  • Sample size
    The number of valid survey responses obtained or needed. Sample size is important when determining whether how well you can generalize your survey results to the larger population. See the example below for more information.
  • Convenience sample
    A sample which is used because it is easy to access, but is not representative of the population being studied. If the target population for your study is college students but you only survey students at Clemson, then your sample is a convenience sample because Clemson students are not representative of all college students. Convenience samples are best used for pilot studies.
  • Census
    A survey of an entire population, with the aim of collecting data from every single individual of the population. Surveys of entire populations at Clemson are rare and only conducted with the approval of the Executive Leadership Team (ELT). Sometimes called a population survey; however a true census will include multiple modes of data collection to achieve as close to a 100% response rate as possible.
  • Response rate
    The percentage of responses collected out of all survey invitations. The average response rate for student surveys without incentives is between 10-20%. If your response rate is low, think about how to motivate survey invitees to take your survey through incentives or improved communication.
  • Completion rate
    The percentage of survey responses that are fully completed. If the completion rate is low, the survey may be confusing or too long. Completion rates of 100% are possible with good survey design.
  • Survey bias
    Bias refers to choices made in the survey design or data collection which result in systemic errors that distort the responses. There are many sources of bias in surveys which can occur at all phases of planning and executing a survey. In particularly, the difference in characteristics between your initial intended sample and those who responded to your survey may result in non-response bias.
  • Incentives
    Incentives refers to tangible rewards (usually gift cards) are offered to survey respondents as a reward for completing a survey. Incentives can be given to every participant or raffle-based. See the Inventive Cards Procedure for more information on how to purchase incentive cards through Procurement for use with your survey.

Example

Let's assume you want to study the full-time, undergraduate students at Clemson. You know from the Interactive Factbook that the full-time, undergraduate student population in Fall 2023 is 22,144 students. After submitting your survey request and receiving your survey sample, you send out 3000 survey invitations. Of those, 450 students open the survey link and begin to take the survey, but only 360 respondents complete the survey.

Your target population is comprised of all 22,144 currently enrolled, full-time undergraduate students at Clemson.

Your sampling frame is a subset of the target population which excludes those students who are under the age of 18 or have requested confidentiality of their directory information. In this example, the sample frame consists of 21,890 students, or 99% of the target population.

Your survey sample is a list of 3000 students randomly drawn from the sampling frame. This random sample is representative of your target population.

If 450 students started the survey, then your response rate is 15%:

450 survey responses / 3000 survey invitations = 0.15 or 15%

But if only 300 survey responses are completed fully, then your completion rate is 80%:

360 completed surveys / 450 started surveys = 0.80 or 80%

It is important to note that when you perform your statistical calculations, your sample size is 300 (the number of complete responses) and not 3000 (the number of surveys you sent out). Furthermore, while your initial sample was represent of the target population, your response sample may not be. Because of your low response rate, there may be non-response bias if one segment of the population is more likely respond.

For more discussion on sample size and statistical significance, please see Penn State's open education resource on Statistical Concepts and Reasoning.

To determine if your sample was large enough, you can use the Qualtrics Sample Size Calculator. You must first determine what confidence level and margin of error is acceptable for your study. A confidence levels of 95% is typical for research, but a confidence level of 90% may be acceptable for operational surveys.

Based on the example above, if your goal was to have a margin of error of 5% and a confidence level of 95% for a population of 22,144, you would have needed 378 responses. However, you can see from the Qualtrics Margin of Error Calculator that your margin of error is only slightly higher at 5.6%.