Maximize Survey Data Quality with Multimode Surveys

In today’s fast-paced environment, survey requests are often one of many communication types competing for people’s time and attention. The decision about how best to engage respondents—including in-person, online, mail, or phone—is critical to promoting survey responses. Instead of settling on a single survey mode, researchers can leverage multiple modes to complement respondent preferences, reduce respondent burden, and manage costs.

What are multimode surveys, and how do they help meet research goals?

Multimode surveys are surveys that leverage at least two modes of data collection. RTI International has collaborated with many clients to refine and employ the best approach to multimode data collection, and we have implemented several combinations of modes to study diverse sets of topics and groups.

For instance, the 2018 National Recreational Boating Safety Survey (NRBSS)—sponsored by the U.S. Department of Homeland Security and the United States Coast Guard—utilized both online and paper surveys to collect data from recreational boaters and registered boat owners, a population that tends to be older and prefers responding to paper surveys.

Since 2004, the National Center for Education Statistics’ National Postsecondary Student Aid Study (NPSAS) has used a combination of web and telephone surveys to gain a better understanding of how students finance higher education; survey findings indicate that respondents increasingly prefer to respond via the web survey.

In 2020, researchers at RTI adapted their study designs to include additional modes of data collection in response to the COVID-19 pandemic. For example, the National Survey on Drug Use and Health (NSDUH)—sponsored by the Substance Use and Mental Health Services Administration (SAMHSA)—has relied on in-person interviews since its inception in 1971. In 2020, NSDUH continued in-person data collection in places where COVID infection rates were acceptably low. At the same time, the RTI team developed and deployed an online survey to address the need to deliver data from sampled areas where in-person data collection was deemed too risky.

The National Study of Mental Health (NSMH), also sponsored by SAMHSA, focuses on identifying the prevalence of mental health and substance use disorders in the United States. NSMH expanded data collection efforts to include a combination of web, phone, paper, and in-person surveys (when possible) to maximize response rates in pandemic-era survey research.

Not all multimode surveys are alike. How do we decide which modes to use?

Although “multimode” simply means more than two modes, in practice these surveys can be fielded in different ways. For instance, one option is a sequential approach that involves asking respondents to use one specific mode when initially contacted. After a designated time frame, one or more different modes are offered to non-responders as the field period continues. Sequential designs tend to minimize costs by encouraging sample members to respond to an initial, less expensive mode. This approach also simplifies the survey decision for sample members by starting with one response mode instead of initially asking sample members to choose between multiple modes.

RTI used sequential designs for the NRBSS—which asked participants to respond online before offering the option of a paper survey—and for the NSMH—which offered respondents a web survey followed by a telephone survey, a paper survey, and eventually an in-person interview (when possible).

Alternatively, some studies take a concurrent approach—which involves using multiple modes simultaneously to collect data. This approach could involve inviting sample members to respond using their choice of available modes or collecting data by matching specific sets of sample members to a specific mode that is most likely to result in survey completion.

For example, NPSAS gives students the option to respond online or by phone throughout the data collection period. In contrast, NSDUH has utilized in-person data collection during the pandemic, where feasible, and relied on web surveys to collect data from the rest of the sample.

What benefits and challenges are there to consider when using multimode surveys?

Multimode surveys come in a variety of shapes and sizes, but researchers don’t just employ multimode surveys for the sake of having options. Using multiple modes has the potential to provide a variety of benefits, including the following:

• Maximizing response rates
• Improving coverage of the target population
• Controlling costs

These benefits extend to surveys conducted in a variety of unique contexts. For example, the National Inmate Survey (NIS)—sponsored by the Bureau of Justice Statistics—utilizes a multimode approach to surveying inmates about sexual victimization; the NIS leverages a paper survey when inmates are identified as too dangerous to complete an interview via audio computer-assisted self-interviewing on a laptop with an interviewer present. Importantly, this multimode approach maximizes response rates and ensures the data reflect the sexual victimization experiences of the most representative set of the inmate population as possible.

Multimode surveys also have significant challenges to consider when deciding whether to use (1) multiple modes and (2) combinations of modes. When multiple modes are involved, surveys must contain the same questions across the modes to ensure comparability of the resulting data.

For example, RTI collaborated with the U.S. Energy Information Administration (EIA) in 2015 to begin pilot testing a multimode, self-administered option for the Residential Energy Consumption Survey (RECS) at the onset of their traditional in-person data collection efforts. Prior to pilot testing, the RTI team conducted extensive cognitive interviewing to shorten the self-administered version and ensure this questionnaire would produce data with measurement properties comparable to the in-person data.

Another challenge to consider is finding a balance between providing options that will encourage responses and guiding respondents to use the mode that ensures the highest data quality.

The RECS National Pilot included online and paper surveys, as well as an experiment designed by researchers at RTI and EIA to test a “Choice +” option for sample members. This option was named for its ability to give sample members a simultaneous choice to respond online or by mail. However, to encourage responses online—where data quality can be better controlled—Choice+ offered sample members an additional incentive for responding through that mode. Choice + achieved levels of representation comparable to in-person data collection, produced the highest response rates among protocols tested (including among older adults who favored the paper survey), and proved to be the most cost-effective option.

Are multimode surveys here to stay?

For now, the answer is yes. RTI has used multimode approaches to study topics ranging from higher education to mental health and to reach populations as broad as U.S. households and as specific as registered boat owners.

RTI’s experience with leveraging different combinations of modes and approaches has enabled many critical surveys to continue collecting data—even in the midst of the pandemic. In addition to equipping RTI with skills to better navigate pandemic-era research, multimode surveys have also proven to be useful in other unique contexts—such as reaching incarcerated populations and other specific groups.

Going forward, researchers at RTI will continue refining approaches to multimode surveys to meet research goals and manage costs in a rapidly changing world. RTI’s work in this area includes keeping abreast of the communications landscape to understand how preferences and challenges for currently available modes may change in the future. RTI’s efforts also involve monitoring new modes and technologies that can lower barriers to participation and maximizing quality and efficiency when used in combination with existing modes.