CHAPTER 2

Survey Administration

Once you have finished designing your survey, getting your survey delivered through the appropriate format and to your desired audience are key elements to truly making your survey a success. To draw accurate and strategic conclusions from the survey results, researchers need to obtain as appropriate and representative a response rate as possible for any survey administered on their behalf. For example, if a researcher distributes an employee opinion survey to a company with 500 employees, but the survey only yields 20 completed responses, the researcher cannot make accurate conclusions about overall employee satisfaction at the company because the amount of responses is too low to be truly representative. Chapter 5 (Sampling) includes a detailed discussion on the number of respondents needed from the population of interest to draw confident and strategic conclusions from the survey results. The focus of this chapter, however, will be to provide a variety of methods and techniques to help develop and administer an effective survey initiative. In this chapter, the goal will be to discuss the various modes, platforms, and applications through which survey researchers administer surveys to respondents to maximize their response rate.

Survey administration has dramatically evolved since the advent of the personal computer, and more recently, cloud-based software systems have given researchers access to innovative applications to send surveys using a variety of modes and methodologies. The evolution of these modes will be discussed in detail to understand what techniques are available to get surveys in front of an audience of interest.

Researchers use the following modes to administer a questionnaire:

• Personal or Face-to-Face Interviews
• Mail and Self-Administered Questionnaires
• Telephone Questionnaires
• Technological Software or Application Questionnaires

These modes can be used independently, or together with one another in a technique called multimode survey administration. This multimode technique involves delivering questionnaires in more than one format and is often used when attempting to survey hard-to-reach participants. In the next sections, we will outline some of the advantages and disadvantages of each technique listed above, and will provide some guidelines for an effective survey initiative using each method. It is up to the survey administrator to determine which of the techniques is most appropriate, given the context of their own survey initiative.

Personal or Face-to-Face Interviews

Before the advent of analog and digital communication, questionnaires were delivered through personal or face-to-face interviews. This technique can be used for both large and small-scale initiatives. For example, in Exhibit 2.1, you can see this technique implemented in 1950 by the U.S. Census Bureau, who went door-to-door to get input from respondents to a questionnaire.

A personal interview provides direct communication between businesses and consumers, where interviewers ask respondents questions in a face-to-face setting. This versatile and flexible method is a two-way conversation between an interviewer and the respondent. Below are some key aspects to consider when conducting personal or face-to-face interviews:

• The opportunities for feedback: One of the main advantages of the personal interview is the opportunity for two-way feedback. This can help provide a rich source of information and a clearer understanding of each question and subsequent answer for both the interviewer and the respondent.
• Probing complex answers: An important characteristic of personal interviews is the opportunity for the interviewer to probe, clarify, or expand answers to standardized questions. Probing allows interviewers some flexibility; however, there are important guidelines one should follow. Most importantly, the researcher should take steps to avoid biasing the respondent’s answers. This can be accomplished by having standardized probing questions available to the researcher. Misleading respondents by asking targeted probing questions to acquire “ideal” or desired responses will pollute the quality of the information acquired and is considered unprofessional.
• Length of interview: Personal interviews can take a substantial amount of time, which can be very costly. If the research objective requires an extremely lengthy questionnaire, then a personal interview may be the only suitable option; however, if the length of the survey is modest, other techniques may be applied. For example, if the researcher is administering a survey that can last an hour or more, a personal interview may provide a rich source of information, while simultaneously allowing the researcher to provide breaks for the respondent, as needed; however, if the survey is only 10 to 15 minutes in length, a telephone interview can be conducted for convenience and to save on time and cost.
• Completion rate: One advantage of this technique is that there is a higher probability of participants fully completing the questionnaire. Face-to-face interviews help ensure participants are consistently engaged, and are answering questions accurately and fully, since the interviewer can course correct in real time, as needed.
• Props and visual aids: Interviewing respondents face-to-face allows the investigator to show the respondent certain visual aids. For example, the investigator can show the subject a new product sample, or clearly communicate information contained in an infographic to ensure the respondent fully understands the visual information before answering a series of follow-up questions.
• Anonymity of respondent: One disadvantage of in-person interviews is that in a face-to-face interview, the respondents’ identity is obviously revealed to the interviewer. These interviews lack anonymity and may cause the respondent to be reluctant to answer truthfully to questions, particularly questions regarding confidential or sensitive information. Social desirability is an additional concern since the respondent may answer the questions according to how they want to be perceived by the interviewer. To minimize this concern, researchers can hire a third party to conduct the interview to maintain anonymity and confidentiality of the respondent, though third-party interviewers still cannot eliminate this issue entirely.
• High participation: The presence of an interviewer increases the percentage of people willing to complete the interview. The location of the data collection and the tone of the interviewer can influence the participation rate. For example, more people are likely to participate at home than in a shopping mall.
• Door-to-door interviews: Door-to-door interviews provide a more representative sample of the population than do mail surveys because you are actually interacting with your respondent. The researcher can confirm from a door-to-door interview that the household respondent is represented from the public title records of each household interviewed. These interviews can also solve the problem of nonresponse, which may occur in telephone interviews; however, they may cause an underrepresentation in some groups. Consider, for example, how difficult it would be to obtain door-to-door interviews in a high-rise in downtown San Francisco. Additionally, interviewing only these specific high-rise dwellers will result in demographic differences compared to interviewing a sample of the general population. It is important to consider the targeted demographics of your survey population, and then, determine the most effective technique for accessing this population.
• Intercept interviews in malls and other high-traffic areas: These interviews are conducted in a shopping mall or other high-traffic areas. They are relatively low cost and a large number of personal interviews can be conducted rather quickly; however, for these particular interviews, the refusal or noncompliance rate is relatively high and the samples may not be representative of the total population since participation is voluntary. Mall intercept interviews can be very useful if the target group is a small, specific group (e.g., female attendees of a business conference). Mall intercept interviews also have the advantage of using visual aids or conducting taste tests.
• Disadvantages of personal interviews: (1) lack of anonymity for the interviewee; (2) characteristics of the interviewer may influence the response; (3) differential interview techniques may be a source of bias; (4) interviewer errors; and (5) cost.

Mail and Self-Administered Questionnaires

Mail and self-administered questionnaires rely on the efficiency of the written word, rather than the skill of the interviewer. Paper-based surveys must be edited and reviewed many times before they go to print, which is costly. Typos or missing questions can also exhaust the survey budget if caught too late and if the corrections necessitate reprinting.

Data entry can be another budget constraint for paper surveys. Returned survey responses must be entered into a spreadsheet or software application for data analysis. Depending on the volume of questionnaires received, you may either manually enter in all responses, or decide to develop a scan form for your questionnaire. As a rule of thumb, if you are entering over 1,000 questionnaire responses, you should be using scanning technology to reduce data input error. Digital or optical magnetic recognition (OMR) scanning technology is very accurate (99 percent)¹ when the questionnaire is programmed correctly.

Exhibit 2.2 shows the difference between a Microsoft Word questionnaire survey versus a scan form:

Outlined below are several considerations that should be made prior to using mail and self-administered questionnaires. While this list is not exhaustive, it does outline some of the major advantages and disadvantages of using this approach and it’s important for survey researchers to consider each before embarking on their survey initiative.

• Geographic flexibility: Mail questionnaires can reach a geographically dispersed sample because interviewers are not required to be physically present at the time the survey is administered. This is an ideal approach when collecting survey responses from interviewees that are isolated.
• Cost: Mail questionnaires are relatively moderate in cost compared to other basic survey methods; however, they are certainly not the cheapest method of surveying as most include a follow-up mailing. Additionally, the more money that is spent on the printing job (e.g., color, paper type, incentive, first-class postage rate), the higher the response rate is likely to be. One way to reduce this cost is to provide presorted first-class mail rates.
• Respondent convenience: Because the respondents can complete the interview in their own time, mail questionnaires can have a greater chance of capturing accurate information and carefully thought out answers; however, if there is an external deadline, participants may fail to complete the survey on time, or they may answer the survey quickly, and therefore, be less scrupulous in their responses.
• Anonymity of respondent: By using this technique, respondents are more likely to provide honest answers to confidential or embarrassing information because they can maintain their anonymity. Such anonymity can and should be stressed in mail surveys as this is a huge advantage of this survey method technique.
• Interviewer’s absence: While the interviewer’s absence provides a benefit in terms of the respondent’s anonymity, it can also be a disadvantage because the respondent cannot ask the interviewer for clarification on the questions being asked. Thus, selective perception (i.e., the process by which the respondent sees what they want to see as opposed to what’s really there) may occur. Additionally, mail surveys lack the opportunity for surveyors to get direct feedback or ask follow-up/probing questions, and the respondent’s answers must instead be assumed to be completed.
• Standardized questions: In mail surveys, the simpler the question, the less likely it is that respondents will make mistakes while filling out the survey; however, once the questionnaires are mailed, it is difficult, if not impossible, to change the format or explain the questions in more detail.
• Time commitment: If time is a constraint for your survey, then mail surveys may not be the best communication medium. The time between the first mailing and the cutoff date is usually six to eight weeks which may be too long. Despite the time lapse though, this may still be faster than conducting personal interviews, particularly in the case of a nationwide study.
• Length of mail questionnaire: A general rule of thumb is that a mail questionnaire should not exceed six pages. If the questionnaire is too lengthy, an incentive may be required to encourage the respondents to complete the questionnaire in full.
• Response rates: The major limitations of mail questionnaires relate to response rates. Boring, unclear, or complex questionnaires get thrown away. There is no guarantee that the intended subject will fill out the form and there is no guarantee that respondents are similar to nonrespondents. In general, a mail survey cannot be considered reliable unless it has at least a 50-percent response rate, or it can be demonstrated with some certainty that the nonrespondents are similar to those who responded. One way to increase response rates is to include some form of personalized cover letter that communicates the importance of the recipients’ response and how much your organization values their feedback.
• Increasing response rates to mail surveys: Research by Don Dillman has extensively reviewed how to increase response rates for these types of questionnaires.² People who feel strongly about or who are interested in the subject matter of the survey, are more likely to respond to mail surveys, which results in less nonresponse error. The following are several ways to increase mail survey response rates: (1) prepaid postage return envelopes; (2) questionnaires with visually appealing design and format; and (3) clear and easily understood questions. Below are some additional best practices that can help increase response rates.
  • Cover letter: A good cover letter should include:
    • A statement of why the survey is important—social usefulness appeal
    • Appeals to the individual respondent—“help us out” appeal and the egotistical appeal
    • Assurance of confidentiality
    • Indication of prepaid postage
    • A description of the incentive as a “reward” for participation
    • Providing a realistic timeframe of how long the survey should take
    • A description of how the person was scientifically selected.
  • Monetary incentives: These appear to be the most effective incentives and are among the least biasing incentives you can offer. A monetary incentive may be offered to the respondent or to the respondent’s choice of charity, and act as a means of attracting attention to your survey.
  • Keying mail questionnaires: An anonymous key number can be used to avoid the expense and inconvenience of mailing follow-up letters to those who have already responded.
  • Follow-ups: A follow-up may include a second questionnaire or may merely serve as a reminder.
  • Preliminary notification: Advance notification, either by letter or by telephone, has been successful in some situations. The optimal lead time for advance notification is about three days before the mail survey is to arrive.
  • Survey sponsorship: The sponsorship of a well-known business research firm or another prestigious organization, such as a university or a government, may positively influence response rates.

Other self-administered questionnaires: Airlines frequently distribute question cards. Similarly, magazines will often include mail-in surveys. Both of these examples represent inexpensive ways of administering surveys; however, respondents of these questionnaires tend to be very different from those who choose not to complete the survey, significantly biasing the data.

Telephone Questionnaires

Evidence suggests that data collected with landline telephone and cell phone interviews using address-based sampling techniques are a good way to get a representative sample (Messer & Dillman, 2011)³. Interestingly, some research even suggests that these mixed-mode telephone interviews can successfully provide a representative sample of the general population and are thus a higher-quality source of data. Telephone interviews can further provide a rich source of data as respondents tend to be more willing to discuss personal topics over the phone than during a face-to-face interview. Below are some key aspects and distinguishing features of telephone interviewing.

• Computer-assisted telephone interviewing (CATI): All the interviews can be conducted from the same location using CATI software. This allows companies to hire a staff of professional interviewers and to supervise and control the quality of interviewing more effectively. By using CATI, telephone responses can be directly entered into a computer with a keyboard operated by the interviewer. Questions can, and should, be very structured and administered in a systematic and consistent way. Another benefit of these systems is they can use an automatic sample selection and an automatic callback, which can be used to aid the interviewer. Additionally, these systems also allow for random digit dialing (RDD). RDD is when the computer system makes automatic dials to landline phone numbers until a number is answered; however, manual dialing is still required when dialing a cell phone number. The federal Telephone Consumer Protection Act (TCPA) forbids survey and opinion researchers (among others) from calling cell phone numbers using an automated dialer. The Federal Communication Commission (FCC) law also requires that all cell phone numbers be hand dialed. Tough penalties may be imposed on violators of this law.
• Speed: A benefit to conducting telephone interviews is that a multitude of interviews can be conducted overnight. One can just imagine the number of calls a call center staffed with hundreds of operators could make in a single evening.
• Cost: The travel time and cost associated with conducting in-person, face-to-face interviews is virtually eliminated with telephone-based interviews, thus making telephone interviewing a much less expensive option.
• Absence of face-to-face contact: It is possible some respondents will feel more comfortable responding to embarrassing or confidential survey questions over the phone than in person. Although sensitive questions may be answered more willingly, the lack of face-to-face contact can also constitute a limitation, as nonverbal cues cannot be communicated or detected. The lack of body language can lead to miscommunication, and can also limit the amount of information the respondent can convey in an answer.
• Cooperation: Calling someone is much less invasive than showing up to their home. Therefore, people are more likely to respond to questions over the phone than they are to let a stranger into their house. Likewise, interviewers may be somewhat reluctant to conduct face-to-face interviews in certain areas (e.g., high-crime neighborhoods) and thus, conducting phone interviews can allow the researcher access to these populations without risk.
• Representative samples: Some individuals don’t have telephones or have unlisted numbers, and because of this, these individuals can’t be surveyed through telephone interviews. Unfortunately, these individuals may also represent certain demographic groups and if this problem goes unaddressed, the sample and subsequent data may be influenced by nonresponse bias. RDD can partially eliminate the problem of unlisted numbers, but the refusal rate tends to be much higher for RDD calls than for calls to listed telephone numbers.
• Lack of visual medium: The lack of visual aids makes some measurement instruments (e.g., a 5-point scaled response question) difficult—and in some cases, impossible—to administer.
• Limited duration: An ideal survey is one that is short, especially from the perspective of the respondent; however, the researcher must balance acquiring all relevant information with the time commitment they’re asking of their respondents. Unless the interviewee is highly interested in the subject matter; the length of the interview should be made as concise as possible. Approximately 10 to 15 minutes is a good rule of thumb for the length of telephone interviews.
• Callbacks: Callbacks are attempts to reconnect with individuals selected for the sample who did not respond to the first interview attempt. Calling the respondent back can help reduce error associated with nonresponse. Callbacks can be expensive if the respondent is difficult to get a hold of, but they do help mitigate the possibility of biased results. As we previously explained, nonrespondents may represent a certain demographic group, and to exclude such a group from the sample could greatly bias the results.

As a survey researcher, you need to evaluate the benefits and drawbacks of using telephone interviews to conduct your survey. One factor to consider when deciding how to collect your survey data is cost. Cost typically has a major impact on deciding how to complete a study. Cell phone samples, for example, can cost 20 to 30 percent more to complete than a landline sample, in part because response rates for cell phone samples are lower than landline samples. A survey researcher has a higher chance of completing a telephone survey by dialing a landline respondent than a cell phone respondent. Remember, researchers must dial cell phones manually rather than use an RDD, which adds time and subsequent cost to each call made. In addition, cell phone samples are more transient, and therefore, have a higher disconnect rate or wrong number for the respondent. A best practice is to make sure your cell phone sample is geolocated through cell tower operation before making outbound calls. This will increase the accuracy of the respondent living in a specific area of interest. A common best practice is to have operators verify the living location of each respondent when completing a questionnaire.

Below, you will find some common results collected when conducting a mixed-mode telephone sample (Exhibit 2.3):

Technological Software or Application Questionnaires

The proliferation of technology has changed the survey landscape immensely. Since the advent of smartphones and personal computers, more and more surveys are being conducted using survey software and applications. This section of the chapter will discuss some of the advantages and disadvantages of using this mode of data collection and will highlight the various features these software and applications provide. Part of the goal of this chapter is to help you review the vast array of features and platforms that are currently available and to determine which characteristics of an electronic survey are most important for you to meet your business and survey needs. Below are some key aspects to consider for survey software and applications:

• Speed: Given that responses are recorded automatically and questionnaires can be sent directly to respondents, electronic surveys offer one of the fastest modes of data collection.
• Accuracy: A well-designed survey allows researchers to accurately record responses and collect them in a way that allows for easy analysis. Like mail and self-administered questionnaires, survey software and application questionnaires still require the same level of clarity in both the question being asked, and in the format of the response. More often than not, the participant will not have the ability to ask for clarification, so transparency is vital.
• Cost: Survey software and applications can cut costs by reducing the time and travel expenses of face-to-face interviews, eliminating the need to print and mail paper questionnaires, and eliminating the need to staff a call center. In this sense, survey software and applications are a cheap alternative to these other methods; however, these methods are not obsolete, as each technique affords their own unique set of benefits. For example, face-to-face interviews provide a greater opportunity for feedback; mailed questionnaires can be sent to demographic populations using publicly available information (e.g., school districts); and telephone interviewers can help ensure survey completeness.
• Methodology: By using appropriate methodological techniques, survey researchers can help avoid introducing systematic bias into a questionnaire. For example, randomizing certain sets of questions can help ensure that the ordering of questions does not systematically bias a participant’s response. With other modes of survey administration (e.g., paper surveys), this dynamic randomization between participants is not possible, so special attention must be paid to how questions are ordered. It is important for survey researchers to have an appropriate understanding of the various methodologies being used, or capable of being used, with the survey they are administering to improve the overall quality of the data.
• Flexibility: If desired, researchers can use survey software and applications to customize each survey directly to a respondent. For example, skip logic is a feature which allows researchers to display/hide a question based on a respondent’s previous answers. This customization allows participants to skip over irrelevant questions, thus expediting the survey time, while also allowing the researcher to expand a specific line of questioning, resulting in more detailed information.
• Accessibility: Given the advent of smartphones and the prevalence of personal computers, electronic questionnaires allow participants to easily access and complete a survey from almost anywhere in the world. This ease of access allows respondents to complete a survey at their own convenience.
• Incompleteness: One of the challenges of using software technology is that participants can inadvertently—or perhaps, purposefully—skip over questions without completing them. Participants can be easily distracted, which can lead to inaccurate/incomplete information; one method for overcoming this challenge is to use an attention check⁴ (e.g., “Please mark ‘Slightly Disagree’”). Additionally, measured steps must be taken to reduce survey fatigue, such as limiting the number of questions asked or using targeted questions to measure a construct of interest.

Keeping these aspects in mind, we will now discuss what considerations must be made when using survey software and applications, as well as how the features offered by different platforms can improve the quality of an electronic survey. Although the temptation may exist to use these platforms to create beautiful, interactive, and stimulating surveys, it is important that features such as these serve to enhance an already well-made survey, and not the other way around. Before you begin searching for a survey platform, consider which features are most important for your survey initiative.

Initial Considerations

• Number of Questions/Responses/Surveys: Many survey platforms distinguish themselves by the number of questions/surveys you can create, and the number of responses you are allowed. It is important to have an accurate estimation of the size of the sample and the total population you will be working with, how often you want to survey them, and how many questions you plan on asking.
• Exporting Results: At the outset of an initiative, it is important to know how you plan to analyze the data you collect. Generally speaking, as the price you pay for a survey platform increases, so too does your ability to export the collected data to additional programs. You want to make sure whatever platform you are using allows you to export data in your desired fashion.
• Data Management: BACK UP THE DATA! BACK UP THE DATA! BACK UP THE DATA! As many experienced survey researchers and professionals can tell you, there are few more easily preventable problems than losing valuable data. Various survey platforms allow you to back up your data in the event of a technological malfunction. Platforms often differ in the amount of data you can store, where the data are stored, and your ability to “clean” the data within the application itself.
• Analyzing Results: Some platforms contain analytic features that allow you to filter your responses based on answers to specific questions (e.g., demographic variables), and some have integrated analytic programs capable of performing various types of statistical analysis. If you do not have access to a statistical software package, the platforms with integrated analysis tools may be of particular interest.
• Security/Password Protection: Most survey platforms offer added security (e.g., encryption of uploaded files) at reasonable price points. Another related feature is the ability to password protect access to the survey itself so that even respondents need to enter a password to complete the survey. This is especially important if you have taken measured steps to develop a high-quality proprietary survey.
• Support: Most survey platforms offer some level of e-mail tech support. For an additional fee, some platforms also include a “prioritized e-mail support” option in which respondents of your survey are prioritized over other survey takers. Other platforms may also provide telephone support. One noteworthy feature of some platforms is the ability to hire a professional account manager who can help design and distribute surveys based on your own unique business needs, thus operating as an on-demand survey consultant.

Survey Software and Application Features

1. Survey Templates: The number of survey templates offered is a distinguishing feature among various survey platforms. This feature is especially important if you are unfamiliar with the types of survey questions you will be asking, and/or if you have a particular type of survey in mind. Various platforms offer different types of templates, as well as different visual themes. Generally speaking, as the price of the platform increases, so does your ability to access various survey templates.
2. Management of Survey/Question Library: Many platforms allow a user to create a database of previously created surveys/questions. Depending on the size of your survey initiative, and how often you plan on conducting survey research, these question libraries can save considerable time and effort.
3. Randomization: Survey platforms offer different forms of randomization (e.g., questions, answer choices, or blocks of questions) at varying price points. Randomization allows you to ensure that the order in which you are asking questions does not have an impact on the way participants respond to these questions.
4. Distribution: Another distinguishing feature of various platforms is how they allow you to distribute your survey. Some platforms offer individual e-mail invitations at lower price points, whereas other platform subscriptions allow you to create a database of recipients that can easily be accessed and managed.
5. Branding: At lower price points, most platforms will include their brand logo somewhere on the survey, and/or in the domain name of an online survey. As the price of a software subscription increases, however, platforms begin to provide “white label” surveys, which allows you to erase unwanted branding and replace it with your own company logo.
6. Compatibility: It is also important to envision how and where respondents will be completing your survey. For example, will your surveys be administered online or offline? Will your survey be completed on mobile devices, desktop computers, or both? And, which Internet browser will your respondents primarily be using? The level and type of compatibility needed for your survey can help you decide between the various platforms and their corresponding prices.
7. Anonymity: To ensure the responses you are getting from your survey are as honest as possible, it is very important to take measured steps to truly protect the anonymity of respondents and to communicate the level of true anonymity to them. Some platforms provide data-scrubbing features that erase individual identifiers and create unique codes that allow you to track respondents without the ability to identify them.
8. Automatic Reminder E-mails: This feature can save you a lot of manual labor and can reduce costs, particularly if you plan on surveying a large population. Most often, this feature is included at medium-tier pricing packages.
9. Scheduling of E-mail Campaigns: Like reminder e-mails, this feature allows survey researchers and professionals to automate the distribution of a given survey by preprograming the time for delivery. This can be helpful if you are conducting longitudinal research initiatives in which you need to survey respondents at multiple time points.
10. Response Validation: This feature ensures your open-ended data responses are accurate, complete, and easy to answer. With this feature, researchers require responses be given in a specific format or characters be limited to a certain amount. This can be especially helpful if specific information is required from respondents (e.g., zip code) and access to the participant is limited.
11. Dashboard: Many survey platforms offer various dashboard features to quickly identify your progress on certain survey initiatives (e.g., completion rate, percentage of unopened responses, time spent per survey). This can be very helpful if your organization is conducting multiple survey initiatives simultaneously and needs a regular high-level snapshot of their progress.
12. Trend Analysis: This feature allows a researcher to track trends in responses over time. For example, if your organization regularly conducts pulse surveys, this feature allows you to quickly identify changes over time in respondent sentiments. Pulse surveys are typically short, quick, and easy surveys conducted at regular intervals (i.e., weekly, or biweekly). These surveys provide a quick snapshot of key company metrics, and can provide insight into the organizational health of a company; hence the name, “pulse.”
13. Logic Features: By implementing various forms of logic into your survey, you can direct participants to specific questions based on their previous responses (i.e., skip logic), reveal or hide questions based on their previous responses (i.e., display logic), or immediately direct participants to a follow-up survey (i.e., chaining). These features can help expedite participation and prevent survey fatigue, as well as provide the researcher with additional information by redirecting participants to a supplemental survey.
14. Piping: This feature allows open-ended responses from previous questions to be “piped” to other locations in the survey.
15. Embedding Media: Certain survey platforms allow researchers to integrate various forms of media into their survey (e.g., pictures, videos, audio files). Although this feature often requires a membership fee, there are certain platforms that provide this service as part of their free subscription.
16. Text Analysis: As software has become more advanced, many survey platforms are offering text analysis (e.g., word clouds, counts). While this feature can provide some compelling data, the complexities of human language can sometimes make interpreting these analyses difficult.
17. Pop-Up/Exit Surveys: By integrating a survey into your website, you can ping visitors of your site with short pop-up surveys, or you can ask visitors to complete a short survey after exiting your site.
18. Collaboration/Accessibility: Depending on the number of individuals on your team, or within your department, some platforms allow users to manage who can access or edit surveys. Most often, at the lower price points, you are limited to one user; however, as the price of the platform increases, you can begin to work more collaboratively. Additionally, some platforms allow an administrative user to allocate which users have access to specific surveys (e.g., the marketing department only has access to marketing surveys, and only your HR team has access to the employee engagement surveys).
19. Language: Many survey platforms allow you to create surveys in multiple languages; however, only some platforms offer this feature for free, while most others require a subscription. This same principal applies to spellchecking features.
20. Integration with other Services (i.e., APIs): An API (application programming interface) specifies how software components should interact. Various platforms offer this feature, but most often this feature is available at a price. These platforms allow users to integrate their platform with other services (e.g., MailChimp, Google Spreadsheets), which allows you to send surveys, and/or record responses, from one platform and integrate them into another software that your company already uses. Additionally, some survey platforms (e.g., Survey Gizmo) explicitly offer the ability to design personalized software for this purpose.
21. Customized Thank You Page and/or Customized Redirect Page: While most often available as part of a subscription, this feature can help offer a personalized touch to thanking respondents for their participation and, perhaps, even increase their willingness to participate again in the future.

Additional common features of survey platforms include: prevention of multiple responses, location information (using IP addresses), timed responses, ability for participants to attach files, automatic question numbering, survey progress bar, save and continue, real-time results to participant, and net promoter score.

Analysis of the Survey Landscape

Given the litany of features listed above and the complexity of the survey landscape, the next section seeks to provide a brief analysis of several survey platforms currently available, and to highlight a few of the useful and distinguishing properties of each platform. In no way is this section intended to be comprehensive; rather, it aims to provide a starting point if you are undecided among some of these platforms based on the needs of your survey.

Cloud-based Survey Platforms

• Qualtrics: One of the early players in online surveys, Qualtrics has most of the features listed above, and is quite a comprehensive survey platform and tool. It is particularly useful for creating highly customized surveys and has many additional professional services (e.g., survey design, customizable dashboards) that can be purchased to help improve the feel and quality of your survey. See Chapter 7 “Reporting Software for Survey Platforms” for a visual example of this platform.
• Culture Amp: Seemingly designed for People Operations or Talent Management departments, this platform offers several “off-the-shelf” survey templates that can be useful for surveying a multitude of Human Resources functions. This platform is especially helpful for novice survey researchers or survey designers as it offers a series of customizable templates, providing a great starting point for designing your survey. Additionally, they also offer several consulting services to assist in the development of your survey initiative.
• Survey Monkey: Another large player in the survey software field, Survey Monkey may lend itself well to organizations that have a dedicated team of survey researchers. Based on their current per user pricing, a small team can direct an innumerous amount of survey initiatives for a reasonable price. Additionally, there are some nice advanced features (e.g., question and answer piping) that are included in a reasonably priced membership. See Chapter 7 “Reporting Software for Survey Platforms” for a visual example of this platform.
• Survey Gizmo: This platform is useful for the enterprising survey researcher. The pricing for this platform is reasonable; they offer a wide range of features, and are willing (for the time being) to build new features, as needed, for an additional cost.
• Question Pro: One great distinguishing feature about this platform is that it offers an integrated data analytics feature. While most platforms allow you to export your data to conduct your analyses, this platform may be a great option if you do not have additional statistical software packages at your disposal. Additionally, they offer text analysis and word cloud visualization to help identify themes in qualitative responses. Finally, they offer some “gamification”-like qualities to help motivate respondents to participate. See Chapter 7 “Reporting Software for Survey Platforms” for a visual example of this platform.
• Survey Anyplace: This mobile-friendly survey platform offers both smiley emoji and star rating options for question responses, which can be a fun way for respondents to answer your questions and are easily understood by participants of all ages. Additionally, they offer a digital scratch card that allows participants to win prizes, based on a chance probability set by the administrator. This platform is both a user and price-friendly option for mobile surveys, and uses “gamification” to incentivize participants.
• Google Forms: This platform is relatively simple in design, allowing the user to create and edit the survey directly onto the survey itself, and has a spreadsheet back-end (i.e., Google sheet) where the data are housed. The platform offers nine question types, branching logic, basic colors and photo themes, and allows the inclusion of YouTube videos, photos, and text. The data are collected onto a Google sheet which automatically displays the data in simple graphs. See Chapter 7 “Reporting Software for Survey Platforms” for a visual example of this platform.
• Snap Survey: This platform offers paper, online, and mobile surveys that populate the same data set, which can be very helpful for managing multimode survey initiatives. Additionally, integrated into their platform is a basic statistical analysis package, which can help reduce the cost of purchasing additional statistical software to run analysis of your data. They also offer printing solutions as well as optical mark and character recognition.

Text Analysis Software

You may have a survey with open-ended response questions that ask participants to provide their answers in a few words or even a few sentences. The analysis for this kind of data is called Content Analysis. Content Analysis is a research method used to make reliable and valid conclusions from textual material. Depending on the number of participants surveyed, this volume of textual information can be cumbersome when it comes time to conduct your analyses. For this reason, the use of certain text analysis software can greatly help.

• IBM SPSS Text Analytics for Surveys: This software allows you to transform any unstructured survey text into quantitative data and gain insight using sentiment analysis. This platform can process a variety of textual information and organize it into meaningful concepts or clusters. An example of this text analysis software is shown in Exhibit 2.4.

Survey Dashboard Reporting

Whether you are collecting longitudinal data, frequently analyzing similar reports, or conducting regular pulse surveys, implementing a survey dashboard can be an incredibly effective way to visualize tombs of information using a simple, easy-to-digest design. This portion of the chapter will focus on the considerations that should be made before creating, designing, or purchasing a survey dashboard.

“A dashboard is a visual display of the most important information needed to achieve one or more objectives which fits entirely on a single computer screen so it can be monitored at a glance.”⁵ When approached purposefully, a well-designed survey dashboard can help quickly identify important survey metrics, allowing individuals to make more insightful and accurate observations in real time. Of course, deciding what information to include, and how to visually represent this information poses a significant challenge to organizational researchers and professionals. When deciding if a survey dashboard is appropriate for your organization, it is important to consider several important questions. Do you need a simple, data-driven visual story for an objective (i.e., is a dashboard necessary to complete your objective)? What metrics will help you most meaningfully track your progress toward your objective? Based on your budget, is it more worthwhile for you to create an entirely new dashboard, or to purchase an “off-the-shelf” design? Taken together, the answers to these questions will help you unpack your specific needs, and think more critically about which long-term solutions are most advantageous for you, your organization, and your survey needs.

Important questions to consider include:

• What are the most relevant metrics?
• What level of specificity is required?
• How can we represent this information in a visually appealing way, avoiding clutter, but also not masking or excluding important information?
• How can I involve stakeholders early in the process to ensure the dashboard meets their needs?
• How can the information contained within the dashboard be personalized for various stakeholders?

Given the complexity of this undertaking, many organizational researchers make costly mistakes. Therefore, careful consideration of the questions listed above is critical prior to making any purchases. For example, after deciding what information is most relevant to include in the dashboard, it is important to contextualize this information for end users who will be investigating trends in the data. This need for contextualizing the information collected should also be balanced with maintaining simplicity, since excessive details can overwhelm the user and make it difficult to parse out the most meaningful and relevant information. In designing and choosing a dashboard, you want to limit the amount of thought required from users to identify the key takeaways and important, high-level information. Often, this can be achieved by providing a macro-level snapshot of the most important information, and allowing users to “drill-down” into specifics, as needed, that can help further contextualize the information.

Another common mistake of organizational researchers is to introduce unnecessary complexity. For example, it is important to limit the amount and type of data and information being visualized (i.e., using a limited number of charts and tables). To decide what types of data visualizations are most appropriate, confer with the stakeholders who will be using the dashboard on a regular basis. This can help highlight the information that is most relevant, and identify information that can be excluded for the sake of simplicity. Design the dashboard according to who will be using it most. Some dashboards can include “Survey Health” features that allow a user to get a quick snapshot of their survey response rates, completion percentages, and other data trends. Although some of this information can be interesting to look at, it may not offer strategic insight into the results of the data, and may, in fact, distract from more important information. Try to limit the amount of useless decoration, and do not misuse, or overuse, color; aesthetically pleasing and simple features allow users to focus on only the most relevant information.

Finally, it is important to keep in mind that a dashboard should not replace a more detailed analysis. While it can, and should, represent critical information at a high level, it is still important to investigate the specifics of the objective in which you are interested.

¹http://www.cis.hawaii.edu/scanning-service/what-is-omr-scanning/

²D.A. Dillman. (1972). “Increasing mail questionnaire response in large samples of the general public.” Public Opinion Quarterly 36, no. 2, pp. 254–257.

³B.L. Messer and D.A. Dillman. (2011). “Surveying the general public over the internet using address-based sampling and mail contact procedures.” Public Opinion Quarterly 75, no. 3, pp. 429–457.

⁴A.W. Meade and S.B. Craig. (2012). “Identifying careless responses in survey data.” Psychological Methods 17 no. 3, p. 437.

⁵S. Few. Information Dashboard Design: The Effective Visual Communication of Data 2006.