On occasion, we are asked about comparison of quantitative research results to norms – average scores pulled from standardized data points, typically drawn from studies or questions of a similar type or within the same industry.

We do not compile data from our (usually custom-designed) research studies to be compiled into normative databases, but in some cases, we can acquire norms from an external resource to be compared against the results from a study we run. However, we generally recommend against such comparison, as any conclusion drawn does little more than pose a statistical difference, easy to misinterpret and requiring a footnote to explain the foundation upon which it is based.

As the research designs we develop for each study are customized, mapping to the strategic objectives of the study, the subsequent survey results are often difficult to compare side-by-side to any norms we can acquire. We generally recommend against application of norms for several reasons:

  • Differences between survey instruments used to collect normative data and those used to collect data for a custom study
  • Disparities between the sample frame used in the studies compiled into norms as opposed to the sample frame constructed specifically for a custom study
  • The enormous range of variation in normative data
  • The adherence to strict integrity of question wording across surveys
  • The lack of integration of norms into a strategic business model
  • The lack of control over the methodology used for collection and synthesis of normative data

We are always willing to consider this type of comparative analysis as a part of a larger analytical plan, but these shortcomings only seem to support our recommendation of a customized research design; if comparison of quantitative data is in order, perhaps a pre/post design, or a benchmarking/tracking design would be appropriate.

Advertisement