Why online rating system doesn't work?

From: Tim Lee <TLee_at_webcmo.com>
Date: Fri, 25 Jun 1999 09:38:02 -0500 (CDT)

An online rating system was designed to achieve two goals:
to measure the traffic on a web site and to provide the
demographic information of the visitors on a web site. Can
such online rating system work toward these goals?

In a recent report, we knew that the actual traffic on a web
site is totally different from the traffic report provided
by an online rating system about that web site? Why did it
happen? Can we make the accurate measures by using the
current online rating system?

>From statistic point of view, the current online rating
system has the following drawbacks:

1. The panel size.
By using the most popular sample size calculation formula,
we need a panel of about 1 million (random sample) to
measure the web site with traffic at about 1% of the online
population and with margin of error less than 0.1% (ie,
margin of error is less than 10% on the web site about its
traffic) at 95% confidence interval.

The smaller the panel size, the less accurate the traffic and
the fewer web sites it can accurately measure.

Suppose the current online population is about 200 millions.
So with a panel of one million people, we are able to measure
a web site with 2 million unique visitors with an error of
(+/-) 200,000. If a web site has only one million visitors
per month, the error is about 20%.

2. Panel sampling: USA vs. International
Remember the Internet is called World Wide Web. But many
times it was treated as a UWW (USA Wide Web). If the panel
is not world wide random sample, it is unlikely to provide
accurate measurement.

But it is almost impossible to get a world wide random sample
currently.

3. Panel sampling: static Vs. dynamic growth
Usually such panel study can only be used to study a STATIC
event. With such rapid growth of the Internet, everyday there
are many new onliners whose online interests and activities
may totally be different from those who are already online.

So even we have an effective way to get a random sample world
wide (and large enough to measure the traffic) and conduct
such measurement, when we get the report, the sample is no
longer representative.

How often do such online rating systems modify their panel
samples? If it is assumed that the population is static, why
do we need them to measure the traffic month by month?

4. Measurement: Statistical analysis Vs. arithmetic calculation
For most online research, the researchers just did some
arithmetic calculation instead of statistical analysis.

What is the difference?

I am using a simple example to illustrate this issue. What is
the difference between a "mean" and an "average"? They are the
same by numbers but have totally different meanings. An average
is a number but a mean is an important measurement of a random
variable.

Unlike an arithmetic calculation, a statistic analysis is to
study a random variable (such as online population) or a
stochastic process (such as the growth of online population).
But how many times have we seen online researchers using the
concept of a statistic analyses to conduct their research? In
this online rating system case, we are studying two variables:
online population and its activities. Therefore, the traffic
on a web site is the product of these two variables. What are
the statistic characteristics of this new variable? No one
tries to tell us. What we get is the arithmetic calculation.

While we don't know exactly how many online, how can we
accurately calculate the traffic on a web site???

But a more fundamental question is: Do we really need to know
the traffic ranking of a web site to make our advertising
decisions?

5. Demographics
There is no report about the errors of the demographic
information provided by the online rating systems. But I
would expect a larger discrepancy. Why?

Except the sampling issues (see above), let's assume I was
one of the panel member of such an online rating system, what
demographic information do they have? Mine! But when my son
uses the same computer to enter an online gaming site, what
do you expect they tell you?

But another more fundamental question is: Do we need such
demographic information to make our online targeting decisions
even it is accurate? For example, if you want to promote a game
related service, where do you want to go? To a web site that
its demos fits your target or a game related web site or a
search keyword "game" return page? Demographic information is
the dominated information for online targeting.

I doubt such online rating system design is able get a "C"
grade in an MBA marketing research class.

Regards,

Tim Lee
Editor -- Journal of Web Marketing Research
http://www.WebCMO.com
Innovations in web marketing research & strategies

========================================================================
------------------------------------------------------------------------
        This week's Online Advertising Discussion List sponsors:
                  Flycast, Streammedia, and TeknoSurf
    
WEB ADVERTISING THAT MAKES YOU A HERO. Our Web experts optimize your
buy across 922 sites. All for less than what you'd normally pay for
placement on one. Call it superhuman. http://WWW.FLYCAST.COM/OA

                                ---

    Sure, you can continue to rub two sticks together.But that's
      not gonna make the Advertising Gods real happy. Check out
    http://www.radicalmail.com for advertising that really works

                                ---

TeknoSurf, the Internet's largest Interactive Direct Marketing company,
   offers opportunities for advertisers and web publishers to conduct
business and generate revenues via the Internet. http://www.teknosurf.com

------------------------------------------------------------------------
========================================================================

Received on Fri Jun 25 1999 - 12:09:23 CDT