Survey Tips

The Definitive Survey Design Checklist

 

In our previous blog, we outlined the basic principles of questionnaire design for writing great, effective survey questions. Of course, your overall survey design will include more than just the questions. You will also need to include a survey title, provide instructions and add a thank-you closing statement, as well as review and test the effectiveness of each element before launch. If only there were a simple survey design checklist to follow to ensure you didn’t miss anything.

Well, you’re in luck. We’ve created a handy checklist of all the essential elements to include in your survey, including style and content considerations. We present to you … the Definitive Survey Design Checklist:

Objectives

____ Your survey objectives have been documented

____ You are clear on how you plan to use the data

Survey Design

Introduction

____ Your survey has a clear title

____ You explain the purpose and importance of your survey

____ An estimate of survey duration and length of questionnaire is included

____ You included a confidentiality guarantee or ask permission to share their responses and identify with whom

____ You provided brief company background information

____ You referred to your survey incentive (if any) and provided information about it

Questions

____ The survey begins with a simple question

____ The questions go from general to specific

____ Sensitive questions appear toward the end of the survey

____ Sensitive questions include a “prefer not to answer option”

____ Demographic data appears at the end of the survey

____ Questions are organized by topic

____ All questions relate to your objectives

____ All questions are simple and concise

____ Jargon, acronyms and technical terms have been avoided

____ Response options include all possibilities, using “other” or “none” as necessary

____ The majority of questions are closed-ended for easier data analysis

____ Directions on how to answer are placed before each question

____ Rating scale questions include the rating scale before the question

____ Rating scale questions include a midpoint answer and an equal balance of positive and negative choices

____ Response options are placed vertically when possible, except for tabulated questions

____ Multiple choice questions display the most positive answer first

____ Open-ended questions are voluntary

Closing

____ You thanked your respondents

____ You offered them the possibility to receive the results (if possible)

____ You provided details on receiving the incentive

Final Review

____ You have sent the survey to colleagues or friends to validate wording and timing

____ You have pre-tested the survey by sending to a small group of respondents in the target population

____ The survey takes less than 10 minutes to complete

____ You have made appropriate revisions to address any issues uncovered

By following this guide, you will be well on your way to survey research success. For more information on survey design or any aspect of survey mail management, contact us today!

 

 

 

By |2019-03-20T10:48:48+00:00December 20th, 2018|Survey Research Services|0 Comments

How to Write Great Survey Questions

When starting a survey project, most people look forward to the fun, creative part of writing the questions. However, it doesn’t take long to realize that writing great survey questions is not as easy as it looks. Questionnaire design is more science than art – requiring critical attention be paid to question and answer order, structure and phrasing to ensure you get the reliable, quality feedback you are looking for.

A simple question, such as “How much did you enjoy the program?” could wreak havoc in your results, because it is inherently biased towards a positive response. “How did you feel about the program?” would be a more effective approach. Other pitfalls include asking multi-part questions, having overlapping answer choices, or asking the more difficult questions too early.

But have no fear. Outlined below are the basic principles of questionnaire design, along with some helpful tips, that will have you writing great, effective survey questions in no time:

Before You can Start Writing Great Survey Questions 

    • Know your objectives. Write down the purpose of your survey scanning, what information you need, and how you plan to use the data.
  • Work backwards. Make a list of the specific answers you need first, and then use that to drive your questionnaire.

Basic Guidelines for Writing Great Survey Questions

    • Keep questions focused. Make sure each question is designed for specific feedback. Avoid double-barrel questions like “How do you feel about our products and services?” as some respondents will focus on products and others on services. Instead, separate them into two questions.
    • Put easier questions first. This will increase participation and establish trust. By getting comfortable with the survey research by answering a few less complex questions first, your participants will be more likely to answer the more complex or sensitive questions later.
    • Organize by topic. Similar questions should be grouped together so the questionnaire flows naturally.
    • Keep it short and simple. Questions should be short, focused, and easy to answer. This will ensure a higher response rate and limit survey fatigue.
    • Be consistent. Use uniform rating scales, word choices and definitions throughout your survey. If you start with 1=low and 5=high, stick with that format.
    • Be precise. Avoid generic answer choices like “sometimes” and “rarely”. Use actual numbers instead (e.g, “more than 3 times per week”).
    • Be balanced. Provide an equal number of positive and negative response options.
    • Be complete. Include all possible answers, and make sure there is no overlap between answer options.
  • Eliminate bias. Try to construct the questions as objectively as possible. Avoid leading questions like, “Can you see why this product was voted best in customer satisfaction?” Instead, ask how they would describe their satisfaction level.

Common Question Types

Survey questions fall into two categories: Structured (fixed response) where they choose from a provided list of answer options and Non-structured (open-ended) where they can fill in their own text or numeric answer. Both are extremely useful, depending on the type of feedback you need.

Following are the most commonly used question types:

Multiple Choice

These are questions with two or more answer options. These are useful for collecting structured responses.

Single Response Style (select one answer)

Example 1: Do you smoke?     Y / N

Example 2:  If yes, how many cigarettes do you smoke per day?

use multiple choice to write great survey questions   

A common pitfall here is missing a possible response. Depending on your question, you may need to add a choice called “none”, or if you would like additional details, you could try an “other” option with space for a written response. You also want to make sure there is no overlap, such as using 10-20, 20-30, etc. in the previous example, which would clearly taint the results.  

Multiple Response Style (you may select more than one answer)

Example 1:  What is your race? (check all that apply)

use multiple response style to write great survey questions

Rating Scales

Rating scales ask respondents to rate how much they agree with a certain statement using a common scale (e.g, 1 to 5, where 1=low and 5=high). These are useful for gauging their opinions, attitudes and behaviors. When using rating scales, it is important to make sure you have a neutral option and a balanced, equal number of positive and negative responses. Scales most commonly use 5 or 7 options.

Example 1: The teacher was knowledgeable.

Example 2:  How would you describe your experience navigating the instruction manual.

use rating scale to write great survey questions (sample 2)

Common pitfalls here include being inconsistent with your scales (leading some respondents to answer incorrectly) and asking leading questions, such as, “We pride ourselves on our easy-to-use manuals. How easy was our manual to read?”

Ranking Scales

These ask respondents to rank a list of items in order (e.g, from favorite to least favorite, or most important to least important). It is recommended that you use these with caution. They are known to be reliable at determining first and last place, but not so much the fuzzy middle, as respondents often have to choose a pecking order for items that are essentially of equal value to them.

Example 1: Please rank the following customer service features in order of most to least important when contacting our agency by phone (1=most important, 5=least important)

use ranking scale to write great survey questions

Open-Ended Questions

These are questions with no provided answers options. Respondents answer by writing in their own text. These are great for eliciting responses about attitudes and opinions in a respondent’s own words, or having them provide a numeric answer without a suggested range. The downside is it requires extra time, can cause survey abandonment, and makes data collection and analysis more challenging.

Example: Name two ways we could have improved your customer experience today?

Questionnaire integrity is critical for getting quality data. By following these tips and guidelines, you will be well on your way to success. 

For more information on question and survey design or any aspect of survey mail management, contact us today!

By |2019-03-20T10:51:08+00:00December 6th, 2018|Survey Research Services|0 Comments

When to Do a Multimodal Survey?

Multimodal or mixed-mode surveys are research surveys that use two or more forms of communication to reach respondents (e.g, telephone and email). In today’s increasingly complex, interconnected world, we now have ways of communicating that didn’t exist even a few years ago. The list of channels seems almost endless nowadays, including regular and express mail, email, online, social media, mobile (text, instant message), scannable paper, telephone, kiosk, tablet, in-person, video, and more.   

So how do you know which channel or combination of channels is right for your survey project? The answer lies in the target audience you are trying to reach (as well as time and cost considerations). For example, older respondents are typically less trusting of online channels and can be reached more reliably by landline telephone and regular mail. Millennials may not even have a landline and would be more receptive to an email or a text message. Teens are generally receptive to mobile and social media. One constant to be mindful of, however, is that everyone has a physical address where they live – making mail a preferred channel in almost any multimodal research effort.

Not only do various target populations have preferred forms of communication, but there are also subsets within them that prefer something else entirely. Mobile phone call vs. text message among Generation X’ers is a prime example, in which someone’s preference is highly personal.

You should do a multimodal survey if you have a target population with varied respondents or hard-to-reach respondents.

Not only can a mixed methodology approach to data collection help you reach more respondents, but it can also help you maximize response rate. That’s because multiple channels give you more opportunities for follow ups, reminders, and options to complete the survey in a format that suits them.

As a leading provider of mail and multimodal surveys, we manage mixed methodology research every day, including planning, production, distribution, fulfillment and data collection. That said, our most popular service by far is Mail to Online. In this strategy, respondents are notified by regular mail and given instructions to complete the survey online. Respondents then have the option to print out or request a paper survey and complete a hard copy or complete the online survey.

Multimodal survey planning and execution requires a high level of expertise to run seamlessly. Your survey provider can help you manage all the details, including which channels to use, along with the projected cost and timeline.

For more information on multimodal surveys or any aspect of survey management, contact us today!

By |2019-03-20T10:54:13+00:00November 21st, 2018|Survey Mailing Services, Survey Research Services|0 Comments

Overcoming Survey Budgeting Challenges

Establishing a budget for your survey project is challenging. There are many facets to consider. It’s tempting to pull a number out of the air that sounds reasonable and then fit the costs for your survey within that pre-determined budget.

That’s an easy way to set your budget. But, it’s far from the best way.

It’s more effective for you to meet your budgeting challenges head-on. Take the time you need to consider your survey objectives and the costs associated with each step of the project. Only then will you be confident that your budget supports the success of your project.

Work Backwards from Your Expected Response Rate
Your expected response rate is an important budget driver. It impacts your sample size as well as the number of surveys you’ll need to distribute to your targeted respondents. As you know, your response rate will vary based on several factors. A few of these include:

• Questionnaire length
• The complexity of the questions
• The importance respondents place on the survey
• Availability of incentives
• The level of personalization

Every survey project is different. However, every project benefits from achieving the highest response rate possible. Here is an overview of a few factors that influence both your budget and your response rate.

Buying Lists

If you don’t already possess a list of targeted respondents, you’ll need to purchase one. The size of your list will depend on how large of a sample size you need as well as your expected response rate.

Printing Survey Materials

The costs associated with printing your survey materials represent a significant portion of your budget. Consequently, printing is a good area to focus on in your attempt to control costs. Depending on the needs of your project, you can make printing decisions that could result in spending as little as five cents or as much as five dollars per survey. How many copies you need to print is one factor. The aesthetics of those surveys is another factor. You can print one, two, three, or four-color surveys. You must also decide on the number of pages, binding, and the weight of the paper. Every decision affects your costs.

Multiple Mailings

Depending on the methodology of your project, it may be beneficial to conduct multiple mailings. Mailing survey packets, follow-up post cards, and replacement survey packets add to your overall cost. You must be careful to balance your expenses with the impact you believe each mailing will have on boosting your response rate.

Consider Offering Incentives
To reduce printing costs and expenses associated with multiple mailings, you should consider offering an incentive to increase your response rate. Research in this area has shown that incentives can double response rates. If you choose to offer incentives, you’ve created another budgetary challenge. First, the price of the incentive increases your budget. Second, research shows that you can expect diminishing returns. So, you must be careful to choose an incentive value that balances cost and effectiveness.

Build Your Budget Around Your Survey
The best approach to overcoming your survey budgeting challenges is to base your budget on the cumulative projected costs of your survey. That’s because you’ll be starting with a clear view of your survey objectives and you can decide how to reach them based on the actual costs of your many options. If you instead start with a specific budget amount, you may find that you’ve picked a dollar figure that bears no practical relationship to your survey objectives. By constraining yourself with a budget at the outset, you shift your focus away from what matters most.

To help you build your budget around your survey, it is crucial that you consult with your survey partner early and often. Your partner will not only provide you with cost information, they will also offer recommendations that will help you save money while achieving your goals.

By |2019-03-20T11:11:29+00:00March 7th, 2018|Survey Research Services|0 Comments

Choosing the Right Sample Size

Choosing the right sample size is vital to accurate survey outcomes. With the right number of survey respondents, your target population will be represented with acceptable accuracy.  Although it may seem like a simple undertaking, you must put a lot of thought into determining your survey’s most effective sample size.

In most cases, the larger your sample size, the more confident you can be that the responses you receive reflect the total target population. However, there is another side to that coin. Deciding on a larger sample size requires you to distribute more surveys, and that drives up the cost of your survey project.  

It’s not as simple as randomly selecting a large sample size and assuming that will equate to successful response rates. The pitfall of using no forethought when selecting a sample size is that you may select a sample that is not statistically significant or will cause you to exceed your budget. To arrive at your optimum sample size, you must start by asking a few key questions.

1. How Large Is Your Target Population?

Choosing the Right Sample Size - How large is Your Target Population

You can begin to understand your sample size needs by first determining the size of the total population of possible respondents. If you want to understand something more large-scale, such as how grocery shoppers are using mobile devices, you will need to use a larger sample size. That’s because there are millions of grocery shoppers who use mobile devices. On the other hand, if you want to assess a smaller, more specific group, such as the opinions of the faculty at a particular university, the population is limited to a few thousand or even a few hundred.

2. How Accurate Do You Need to Be in Choosing the Right Sample Size?

Because you are only using a small sample group out of a larger target population, you must now make some decisions about accuracy. Your accuracy will depend on choosing an acceptable margin of error and accepted confidence level.  

Your margin of error measures how well the responses from your sample reflect the opinions of your entire target population. A commonly chosen margin of error is 5%. You must also keep in mind that lower margins of error require a larger sample size.

Your confidence level represents the degree to which your sample is an accurate sample of the population. If you decide your project requires a 95% confidence level, you are declaring that if you repeated your survey, you would get the same results 95% of the time. Higher confidence levels require a larger sample size.

3. What Is Your Expected Response Rate?

Arriving at an answer to the first two questions is relatively straightforward. First, determine the size of your target population. Then decide how accurate you want your survey results to be.

Now, on to the challenging part: estimating what your response rate will be. Estimating your response rate is often difficult; there are many factors that contribute to a respondent’s decision to complete and return your survey. The list of factors includes:

    • Questionnaire length
    • Question complexity
    • Importance of the survey to the respondent
    • Level of respondent interest in the survey
    • Availability of incentives

Fortunately, you are in control of most of these factors. Depending on your target population and the decisions you make about how you will administer your survey, your response rate can range from 1% to 60%. Lower expected response rates require a larger sample size.

For more tips on Survey Design, check our blog post on How to Write Great Survey Questions

Working with an Experienced Survey Administration Partner in Choosing the Right Sample Size

Your sample size has a direct impact on the validity and cost of your survey. Consequently, it’s crucial that you get that number right. Once you have determined the right sample size for your survey, there’s still more work to do. How will you distribute the surveys? How will you gather them and analyze the outcomes?  

This is where your survey administration partner can add value to the process. When you work with an experienced vendor, they can consult with you about the nature of your survey research and suggest choices that will help you achieve your goals in an efficient and cost-effective manner.

Your partner can help you measure your target population and decide on an appropriate margin of error and confidence level. In addition—although no two survey projects are ever exactly alike—your partner can rely on their experience with working with a wide range of organizations and projects to help predict what your response rate might be.

When you are planning your next survey project—and trying to understand your optimal sample size—consider getting help from an experienced survey administration partner.

If you’d like more information on incentives, project consulting services, survey research services, incentive fulfillment or survey mailing services in general, contact us today!

By |2019-05-24T19:47:16+00:00March 6th, 2018|Survey Research Services|0 Comments