Today we continue from previous posts where we have discussed the first and second of three phases of survey research where guidelines should be implemented:

  1. Determining a Survey Sample Universe
  2. Creating the Survey Instrument
  3. Collecting, Analyzing, and Reporting the Data

This time we will discuss the third phase.

There are guidelines to follow when conducting a survey and analyzing and reporting the resulting data. Once the questions have been finalized, the survey can be administered. Following the guidance of the academic literature and to avoid any potential bias being introduced into the surveys, we often outsource this activity to an independent firm. The firm then administers the survey, collects the survey data, and sends us the survey data.

When analyzing the data from the survey respondents, a researcher should check for non-response bias, inadequate responsiveness, and respondents’ levels of understanding the task. Non-response bias simply means that a large portion of the sample universe did not respond to the survey, and the literature is mixed regarding whether non-response bias actually affects reported results. Inadequate responsiveness refers to when a large portion of those who did respond to the survey failed to provide an answer to a given question, particularly to the main valuation question. If any of these problems arise, it may suggest that the researcher needs to perform additional data analysis. As one measure of data reliability, cross tabulations can be run on the data to determine if individuals from certain age groups, income levels, etc. provide similar value estimates. Often there are nuances in the survey data that are not obvious simply by looking at the descriptive statistics of the survey responses. To determine value estimates, a researcher may experiment with different econometric models until one is found that best matches the data and eliminates potential bias.

Once we receive the results of the survey and analyze the data, we then must report the results (see the figure below for one example). To report the results thoroughly, the guidelines suggest that researchers provide detailed information about their research process and their methods for analyzing the data. This includes everything from reporting how the survey sample was collected to what kind of econometric model was used to calculate values. Diamond’s 2011 “Reference Guide on Survey Research”[1] is helpful in this stage because it outlines all of the items that a survey report should generally account for. Those include items such as the purpose of the survey; a definition of the target population; a description of the sampling frame; a description of the sample design, including the method of selecting respondents, the method of interview, etc; a description of the results of sample implementation; exact wording of the questions used; a description of any special scoring; a description of any weighting or estimating procedures; estimates of sampling error, where appropriate; statistical tables clearly labeled and identified; and copies of interviewer instructions, validation results, and code books.

These few examples describe the degree of detail involved with analyzing and reporting survey data. For more information, feel free to use our Ask an Expert page for any questions.

– Abigail Mooney and Sarah J. Kilpatrick

1. Diamond, Shari S. 2011. “Reference Guide on Survey Research.” pp. 359–424 in Reference Manual on Scientific Evidence, 3rd Edition. Federal Judicial Center. Washington, DC: National Academies Press. ↑