And a selection of preferred statistical techniques…

And, of course, what keeps us all up at night?

Here is the PowerPoint presentation created by Next Gen Market Research that explored all things market research from the inside out.

Social media implantation surveys are a way of reaching potential survey participants while they are interacting with online sites and may have a few minutes to share an opinion. Given the foreknowledge of where to locate potentially millions of survey participants, social media sample collectors have crawled out of the woodwork after building up their databases. As a social media sample collector we recently met with explained, many internet gaming sites require gamers to pay with virtual money or points to play games online. Sample collectors realized this fact and have teamed up with gaming sites to offer virtual dollars for online surveys. The survey is seamlessly implanted into the participant’s page and can be accessed through a quick link. After completing the survey, participants get paid in virtual dollars and continue their gaming experience instantly.

The cost benefit – virtual world means virtual money, meaning costs in the real world are typically lower than in the average quantitative roll-out. The drawback here is simply this – these methods capture mostly general respondent types, meaning “a woman” or “an 18-year-old who uses Facebook”. The depth is lost when trying to reach buyers who may have more complex behavioral or psychographical reasoning for purchasing a product or service. Participants with more than one or two unique qualifications such as “moms who work at Wal-Mart and have two kids” will still have to be recruited intentionally by a skilled sample sourcer.

When it comes to mobile phone surveying, the benefits and pitfalls are split almost equally. The sample is usually gathered from a company database, eradicating the middle-man. This manner of cutting corners is a time saver to surveyors, but is going to be an issue for sample collectors who rely on old-fashioned methods to gather survey participants. Another downside to this type of survey is that, yet it is ideal for quick polls and opinion surveys, more logical or lengthy surveys are just not practical. So far the most advanced method of collecting and analyzing this data is by way of exportation to an Excel file. All in all, these two methods may make for a quick and dirty source of information, but ultimately they still serve as a companion to in-depth surveys that require hand-chosen participants. The last thing we researchers want to do is skimp on data quality even if it means getting results a little bit slower.

Try it out, if you call 301-763-4636 you can hear bits on information such as how many Americans have dogs or cats, how many shelled peanuts we eat each year, a 152 year old patent for pencil erasers, etc.

The Times article has some of the scripts and recordings for those of you who don’t want to call and have to ask to be put on hold (or you an call the line dedicated to playing the messages without prompts or talking to people (301) 763 2222).

Any one who has read Freakonomics will be familiar with this story: A University of Chicago grad student does some research in the projects and ends up with a bunch of data about the finances of crack dealers. As it turns out, that was only the tip of the iceberg.

In Gang Leader for a Day, that grad student, Sudhir Ventakesh, tells the whole story. It all begins as a little quantitative survey conducted for one of his faculty advisors. It becomes a sprawling ethnography that takes place over the course of five years, against the wishes of his mentors and his own best judgment. Along the way, Sudhir sees the world inside the roughest projects in Chicago unfold before him in all its complexity. We meet a gang leader who deals crack and serves as a sort of social glue for the community. We meet a community leader who does what she can to help her fellow residents, all while taking kickbacks to better her own circumstances. We meet the police, charged with patrolling the projects, who never show up when called.

Ultimately, it’s a great piece of research, and a story well told.

We’ve found mobile survey research very useful in the past for studies that are aimed at B2B targets, or highly mobile and/or tech-savvy consumers.  With the growth of smartphone and netbook usage, particularly over the past couple of years, this methodology seems more and more vital.  It’s great to be able to connect with research participants while they’re on the go, or engaged in their lives (whether for work or play) – not just when they find time to check their email or call back for a phone interview.

We find that expertise in both questionnaire design and data management is absolutely necessary to do in-depth analysis on data collected through this medium. It’s important to be concise in wording your questions and answer options, and you need more than just quick polling data to illuminate insights that lie beneath and behind survey responses.  But that’s why we’re here – to recommend a technology solution if and when it’s appropriate for a research initiative, and to exercise our skills and perspectives to take the strategic insights to the next level.

We try to find opportunities to include MaxDiff exercises in our studies that focus on consumer preference.  We’ve found this technique particularly useful in segmentation studies where key drivers and motivations for behaviors are of interest.

Including a series of MaxDiff exercises adds to the overall survey length, but this analysis can be very helpful in illuminating how consumers truly prioritize attributes (or features or statements).  MaxDiff exercises allow respondents to rate the importance of attributes relative to other attributes, rather than independent of them. The end product is a ranking of attributes that can be used to corroborate key insights gained from Likert scale ratings and factor analysis.

Sometimes the results simply confirm consumers’ stated preference, but often, as shown dramatically in this presentation, a re-prioritization emerges. This is a great way to include survey metrics that enable the analyst to approach the results from an additional perspective. So often, a consumer will rate multiple attributes as about equally important, and through MaxDiff analysis we can sometimes see if some differentiation underlies their response.

There’s also an entry on WikiPedia for Maxdiff, but for more information, contact us!

For these types of projects, smart study design is critical, but for the most part these analytical techniques are applied post hoc, in interpretation of trends and spikes that emerge across a large numerical data set. We love this type of work, but we only recommend such an approach if our client’s overall strategic and specific research objectives seem to call for it.

Sometimes, marketing research objectives are best addressed not so much through application of post hoc analytical techniques, but on the front end of the project – through development of a direct, customized, in-depth line of questioning.  This can be true for quantitative or qualitative research, really. This post is mostly inspired by some questionnaire development work I conducted recently: our client’s objectives boiled down to a need to understand specific consumer behaviors and to identify specific factors (both cognitive/rational and emotional/irrational) that went into decision making on the research topic.

We have a large sample for this project, so we’re sure we’ll be able to do some interesting post hoc analysis as well, but at this stage it seems we’re going to be able to provide real insight to our client because we started off in search of specifics, rather than through use of standard survey metrics.  Development of customized questions with detailed answer options made for a smarter survey instrument. We do often add standard, academically validated question formats to our surveys, but some studies simply require a deeper dive. This seems particularly true for technology and health care research, but could potentially apply in any category.  When I start a new questionnaire, I remind myself to watch my step but not hesitate to go down the direct questioning path.

On occasions where I want to find data to support rationale for recommending an online survey methodology for a particular target I consult a number of regularly updated sources. One very helpful source, especially when the targeting is based on demographics or life stage, is the Pew Internet & American Life Project. I’ve referred to their 2005 research on Internet usage many times, but I expect to refer to their newest report on generational Internet usage in the coming weeks and months.

The report views Internet use by Howe & Strauss’ generational labels, but breaking Boomers into younger/older categories. I personally don’t like to hinge my recommendations on generational labels as I’m inclined to abstract generational definitions in favor of softer life stage and behavioral categorizations. These numbers still work as support stats for me, though, as they are largely examined by age divisions; the generational profiling is more precisely applied in their reporting of online behaviors.

I find the Pew reports to be particularly helpful in providing data on the percentages of Americans of different age groups online, and in illuminating what activities persons of each group are doing online (who is engaging in social networking? who is still using email outside of the workplace? shopping online? looking for health information?)  I do not, however, look at this source as the final authority, however, as alas, though this research seems to have been conducted appropriately and with a large sample, the respondents were engaged through landline telephone.  This may help prevent online usage bias, but the problem remains: what trade-offs do we make in choosing an online versus telephone methodology?

We do not compile data from our (usually custom-designed) research studies to be compiled into normative databases, but in some cases, we can acquire norms from an external resource to be compared against the results from a study we run. However, we generally recommend against such comparison, as any conclusion drawn does little more than pose a statistical difference, easy to misinterpret and requiring a footnote to explain the foundation upon which it is based.

As the research designs we develop for each study are customized, mapping to the strategic objectives of the study, the subsequent survey results are often difficult to compare side-by-side to any norms we can acquire. We generally recommend against application of norms for several reasons:

• Differences between survey instruments used to collect normative data and those used to collect data for a custom study
• Disparities between the sample frame used in the studies compiled into norms as opposed to the sample frame constructed specifically for a custom study
• The enormous range of variation in normative data
• The adherence to strict integrity of question wording across surveys
• The lack of integration of norms into a strategic business model
• The lack of control over the methodology used for collection and synthesis of normative data

We are always willing to consider this type of comparative analysis as a part of a larger analytical plan, but these shortcomings only seem to support our recommendation of a customized research design; if comparison of quantitative data is in order, perhaps a pre/post design, or a benchmarking/tracking design would be appropriate.