by Edgar A. Kully, Principal - Crestwood Associates

There has been a lot of discussion lately about the best software for conducting e-mail surveys. There have also been some cautions about this vehicle. It is true that email surveys can be an inexpensive way of gathering data, but you need to be aware of what that data really means before taking any action.

Here's a real life example that we recently experienced: A funny thing happened at a very large company that was pursuing the idea of distribution of a new product line through the internet. The product managers introducing the new product knew that they needed to procure funds to move the marketing effort forward. They had a brainstorm. "Let's conduct the feasibility study by conducting the necessary research online, after all this product is going to be distributed online so why not match the research methodology to the distribution channel?"

All this seemed to make sense, so they promptly moved forward with a survey they administered through their own web page. The amount of information gathered amounted to quite a large set of data when someone finished the study. Even better was the fact that several THOUSAND people filled out the survey online. This was GREAT! What could be better? A great amount of information, gathered from a greater number of people than could ever feasibly be done through traditional methods was able to be collected. And best of all, the research hardly cost a thing.

There was a great deal of excitement. Who could argrue with such a large sample size completed in such a short time frame and at almost no cost?

The study showed that demand for this product was exceptional, almost 60% of the individuals responding to the study said that they would buy the product at the price point necessary to gain extremely profitable margins. This was great!

Well, management was less than overwhelmed. Being natural skeptics (and some might say that say modern ludites), they turned down the request for financing until a research methodology that they were used to seeing was applied to the market opportunity.

The research company determined a method of conducting probability sampling among the target market so that a demand forecast could be achieved. A similar type of questioning transpired with only 300 (no more and no less) respondents selected through probability methods and conducted through telephone interviewing. And yes, it did cost some money. However, the difference in results was amazing!

From this probability sample which followed known methods for gathering data, it was determined that demand for the product was really present in only 15% of the individuals at the price point necessary. What was more, demand for this product was shown to not be present among respondents until a year from the time period of the study due to market changes which were taking place in the environment.

Needless to say, the results of this study changed management's opinion of both the product and any research conducted using online methods of testing the online market. The same can be said of any method of testing, if the sample is collected ONLY from those groups that are able to approach because they identified themselves online.

Why is this? We call it the "PYRAMID" effect. Envision a pyramid which gets smaller as one reaches the pinnacle. Imagine that the pyramid is under construction. While there is a great deal of mass in a total pyramid, a quality control sample of the pyramid while in construction would never be conducted on only the top portion of the pyramid. In order to make estimates regarding the quality of construction the pyramid must be sampled throughout the building process and in many different places throughout its structure. No quality control person would ever sample only one point in the process or only one area of the pyramid and attempt to draw conclusions of the quality of the entire pyramid.

While the story regarding the company facing a product decision is true, there is ample evidence to support what this company discovered the hard way. Research sampling of any online population taken only from those willing to respond or give their personal contact information over the internet WILL RESULT IN WRONG CONCLUSIONS. It is just that simple. It is the equivalent to sampling only the top of the pyramid and trying to draw conclusions of the entire pyramid.

Researchers for years have known that to deduce true market information from any other method than one which allows for a true probability sample will result in skewed and misleading information from which wrong directions for strategy and tactics will be indicated. It's the old garbage in, garbage out theory. If the start of the process is not done correctly, skew only gets greater as one follows the information and decisions downstream.

However, the internet IS enchanting. It imperviously seduces even the most stringent research professional at times with its ability to gather large amounts of data from a great number of people at very, very low cost. DO NOT BE FOOLED. As my father used to say, "a person doesn't become a cow simply because they walk through a barn." Large sample sizes, efficient data collection costs, and the size of the database do not make up for poorly constructed research theory at the outset. Let's not check our brains at the door on this one, your business will be the one to suffer.