Our belief is that surveys will need to be broken into bite-sized chunks and that those chunks will likely be completed by different respondents. We have been called heretics; an angry mob has gathered, yet a better solution has yet to be put forward. We are striving to refine our data fusion techniques to enable us to piece together the independent samples into a common sample.

We presented a paper last week at CASRO that discusses our research into this topic. We conducted a study in the US that tested how modular survey designs work with online, mobile web and mobile app-based surveys. We found that chunking surveys showed significant benefits, such as enabling us to be agnostic to the data collection platform and therefore device inclusive. We were also able to make more efficient use of our sample by making use of partial data from incomplete surveys and helping to increase respondent satisfaction. Some of our findings include:

• Respondents complete all chunks. There was a tendency for respondents to complete all the survey chunks in one sitting, even though they understood they had the option to stop and start later. This was true even with the longer survey for both CAWI and mobile app respondents.

• Respondents like the idea of chunking. Nearly half the respondents said chunking made the survey easier to answer, 43 percent said it made it easier to manage their time.

• Our respondents are mobile ready. 80 percent of those who have a mobile device indicated willingness to use those devices to complete a market research study. This was underscored by verbatim responses that thanked us for providing the mobile web options.

• Generally, no difficulties were noticed in answering any of the question types. However, we used tight quotas and a mobile-friendly survey design. Without those controls, we likely would have gotten very different data. Mobile respondents – both mobile web and mobile app – did note, however, that they found typing difficult and time consuming, particularly on certain mobile devices.

• Making bite-sized chunks requires a seasoned researcher with the expertise to think through the survey logic to ensure that all questions that are dependent on other questions are grouped within the same survey chunk.

• Randomizing survey chunks, i.e., positioning chunks in different, random order in surveys delivered to different respondents, can eliminate the Fatigue Effect that is often seen with certain question types late in a survey. However, it makes the design process even more complex because you do not know the order in which chunks are asked, and therefore cannot pipe in answers.

Tapping Chunking’s Potential

Chunking surveys has a great deal of potential, and we are just beginning to delve into its many facets. One topic of focus is identifying nodal questions and using them to indicate which questions are important to have answered by all respondents, and which can be asked to a subset of respondents. Data fusion techniques help us identify these nodal questions and they, in turn, act as connective tissue to enable us to fuse together data from different respondents to make a whole.

Also, we are looking into fusing data together to make use of partial data in an across-respondent design. Data fusion and imputation techniques allow us to either fabricate a whole data set from a partially complete survey or to match two partially completed surveys together to make a whole. Here, we focus on the need for hooks that can link data to the results. These hooks provide direction as to how to connect modules of questions together. Fusion may not be appropriate for all situations, however. In our view, at least 100 respondents are needed to get a reasonable data fusion for a brand. Therefore, for high awareness or high incidence groups, the exercise could reduce the overall sample requirements, but for lower incidence groups it may do the opposite.

We also believe that survey modularization and module randomization has application even without data fusion. Respondents expressed a significant preference for the modularized survey approach because it gave them more control over their time and more options for how they complete surveys. In many surveys, not all questions need to be asked of all respondents. This process provides an elegant way to set target sample sizes for each piece of the study, thereby lowering data collection costs and reducing average interview length.

Based on the research we have done so far, we are intrigued by the promise that both survey chunking and data fusion hold, and we believe these, both individually and together, are topics worthy of further research and innovation.

For more information, click here to download the paper, “Modular Survey Design: A Proposal for Bite-Sized Chunks,” by Frank Kelly, Global Director, Lightspeed Research, Alex Johnson, Director Innovations Group, Kantar Operations, and Sherri Stevens, VP Global Innovations, Millward Brown .

1. “I don’t feel I am getting enough out of my tracking study.”

In our consulting work with clients, this is among the most common reasons for doing a tracking study review.  Data the study was originally designed to collect may no longer be of strategic relevance to the company, and so the project loses its value.

In this case, there are three things you can do:  1) Trim it back to be more cost-efficient and focus on the most important key metric, 2) Re-engineer the survey to ask fresh questions that might be of more strategic value, or 3) Scrap it completely and start fresh.

2. “My tracking study is getting too long and unwieldy.”

Like many projects that evolve over time, tacking studies often get bigger and more unwieldy as questions are added each year. If you find your study has gotten out of control, you may want to look at each component and determine its value to the stakeholders across your organization. We regularly work with clients to conduct comprehensive audits to decide which questions should be removed and which should not.

3. “My tracking study is costing me too much money.”

To reduce costs, you can shorten your survey, optimize your sample, or both. There are ways to ask questions in more effective and efficient ways, which can allow you to get better data while also reducing sample sizes. And there are a variety of sampling methodologies that can reduce the overall cost of sampling.

4. “My tracking study is very boring to complete. I fear this reduces the quality of data it produces.”

This situation plagues a number of our clients, and it is the focus of a tremendous amount of our work. In fact, we specialize in techniques for creating engaging surveys that produce high quality results.  From changing the way questions are phrased, to integrating imagery and interactivity, to introducing gaming elements into surveys, there are a great many options available to clients to remedy a boring survey.

5. “I can gather some of the same data more efficiently in other ways.”

Today there are many new capabilities from social media and data mining that make additional streams of data available. Incorporating these into the mix can transform the role of tracking studies into a bridge to calibrate and understand data from these new sources. As this occurs, we will see more collaboration with text analytics companies to blend text analytics techniques into traditional tracking studies. This means asking less traditional quantitative questions and more open ended questions.

Jon Puleston, Vice President of Innovation, leads the team that provides Lightspeed Research clients with a consultative full-scale review of their tracking studies, exploring areas for improvement in both structure and design.  For more information contact Lightspeed Research at www.lightspeedresearch.com.

Firstly, the final barriers to wholesale acceptance of online data collection are coming down. It was only comparatively recently that clients were wary of using online research outside the major developed markets – citing internet penetration, historic trends and relatively cheap, people-driven fieldwork as the reasons. Over the past two or three years this has completely changed. Those same clients now want consistency and reliability in their data collection and are pushing more and more of their work towards digital data collection. I say digital rather than online because, perhaps ironically, it’s in those emerging markets where online penetration is still patchy or skewed that we are moving from traditional face-to-face or CATI collection directly to mobile surveys. As our clients invest more in their own digital marketing activities and consumers spend more of their time and money on digital platforms, these then become the natural place to conduct all types of research.

So the second trend we are seeing is the true embracing of the digital environment as the place to talk to consumers. Today we can access our respondents online, via mobile devices and through social media platforms. We can survey them, invite them to online focus groups or custom communities, track their internet behaviour and relate their social media opinions to our questionnaire results. All this requires permission of course, and is one of the reasons that panels will be with us long into the future.

Finally, either because of the economic climate or the globalisation of business, international clients are becoming much more global in their outlook. They are moving away from having individual markets buy and run their own local research projects towards a more joined-up research programme. This is challenging for both clients and agencies as the balance that clients need to be able to strike is how they get the efficiencies of scale in purchasing research globally without losing the benefits of knowledgeable, local researchers.

To read the full article on research-live.com, click here.

Accordingly, the topic of question design is vast and multi-faceted, and has been the focus of a great deal of our research on research.   Over the past several years, we have published several papers about the many question design techniques and their impact on research results.  Our most recent paper, Dimensions of Online Survey Data Quality: What Really Matters?, studies this factor through two large-scale multi-country survey experiments interviewing more than 11,000 respondents in 15 countries in a treatment versus control group approach.  Our goal was to understand the impact of question design, as well as other factors, on the quality of survey data across countries.  This work focused on the relative effects of different techniques on answers.

We looked at several aspects of data variance due to question design, including the use of imagery and iconography, the presentation format of questions, and respondents’ motivation to answer in response to question wording.

Imagery-Based Variance

Using images to support choice selection in surveys has significant benefits, allowing researchers to communicate concepts more effectively than using words alone.  However, it has an impact on data that is equally substantial. In our experiment it introduced a data variance of 34 percent, which compared to other forms of variance is very high.

Our paper presents examples that demonstrate how the choice of specific images impacted the results of specific surveys.  The use of imagery improved recall scores on occasions by upwards of 50 percent. However, it must be noted that in some cases it weakened them due to the literal interpretation of the image. The bottom line is that using imagery in surveys without calibration can introduce data variance effects that are overwhelming, and therefore must be carefully implemented.

The use of icons, on the other hand, produced less than 2 percent data variance but had the ability to reduced speeding and speeding data variance as much as 20 percent.  In addition, there was an increase in the consistency of responses across countries.

Dynamic Format

We have also explored the impact of asking questions in generally more interactive formats, such as dragging and dropping, flag sorting, star rating, gambling, list building and various forms of dynamic animated grids.

Only two formats in this experiment delivered measurable data variance. The first was the gambling methodology, where respondents were asked to bet imaginary money on various choices (Details of this technique can be found in The Game Experiments). Gambling encouraged respondents to be slightly more circumspect, and thus shifted the balance of data minimally.  The second format that caused a significant difference was list building.  This method requires further investigation, but the explanation seems to be that respondents tend to sort things into even groups which they may not naturally do when evaluating in a monadic fashion.

Generally speaking, we found that in every case these more creative questioning methods reduced straightlining and increased data granularity.

Question Wording

How questions are worded can greatly impact how they are interpreted. In our experiments we tested wording styles that asked respondents to imagine themselves in certain roles and situations, thus creating a personal connection from the respondent to the question. Our paper documents several examples, all of which demonstrate that this technique evokes a higher volume of feedback as well as higher levels of respondent engagement and satisfaction.

However, these examples also serve to illustrate that question wording is not only a critical part of the process of survey design but also a creative skill of the researcher.  It is wholly dependent upon the subjects, goals and context of the research study, and thus, does not conform to a set of rules, standards or best practices. The most important thing is to be conscious of the impact of question wording on data outcomes, and to make frequent use of pilot tests.

This is part 5 in a series. If you would like to read more, visit:

Part 1: So Many Variables, So Little Time:  A practical guide to what to worry about when conducting multi-country studies

Part 2: Vive la Difference, the impact of cultural factors on multi-country studies

Part 3: Pants on Fire: Testing Untruthfulness

Part 4:  Getting Nowhere Fast: The Impact of Speeders on Multi-Country Studies

In the example below, the first column shows what Adults 18+ look like according to the U.S. Census.  The Category Users column provides a profile of what users of the category look like.   The users of this particular category are more likely to be male and skew younger.

If you set the hard quotas in your survey to force the users to look like Census Adults 18+, you will have a hard time getting the older adults and females because they aren’t as likely to be category users.  More importantly, your sample won’t be representative of category users because you are forcing it to look like the U.S. Census.  It is inappropriate and inaccurate to set quotas to census in surveys with a target sample.  Forcing the sample could actually produce misleading data that may result in a bad business decision.

Let’s continue with the example above.  Clearly, in this category, there are more male users than female users.  If we test a product from the category, purchase intent is significantly higher among males than females.  If we force the proportion of males and females to mirror census (49% male and 51% female) we get a top two box purchase intent of 46%.  If the proportion of males and females is set to be representative of category users (63% male and 37% female) then we get a top two box purchase intent of 53%.  This difference is going to be statistically significant in most cases and could mean the difference between moving ahead and not moving ahead with the product.

This example clearly shows the impact that sample design can play on results.  It is extremely important to make sure your survey completes are truly representative of your target group.  This is the only way to draw accurate conclusions and recommendations.