Re: Tracking and Looksmart

From: Alexis D. Gutzman <>
Date: Thu 25 Jul 2002 11:59:20 -0500

I had said:

> > If you need to be absolutely sure that only spiders make it
> > to the protected pages without getting the form, you can use server-side
> > detection to identify the footprint of the spider (there aren't that
> > many that really matter) and let the server show these agents the
> > content behind the barrier without seeing the form.

Steve Werby replied:
> I think your reply and my earlier reply to Jill crossed paths, but I want
> to reiterate that this isn't foolproof. If you know what user agents most
> search engines regularly use and you know how to set your user agent, then
> you can trick sites that don't know better into thinking you are a speciif
> search engine. In fact, that's a common technique used to view content,
> including meta tags, which competitors publish for specific search
> engines.Plus, many search engines have caught onto the specialized
> content/tag game
> and are now indexing, at least in part, with different user agents and
> sometimes different IP ranges to determine if sites are using this
> technique. If you combine user agent and IPs it'll be a better solution,
> though coming by the info. required to do that will probably be difficult.

You are, of course, correct that combining the two is better, and there may
be ways that SEs try to confirm results by sending different agents out from
different IP addresses. And yes, of course, agent footprints can be spoofed.
However, a competitor (in this example) could just fill out the form as
Elmer Fudd and see the content that's currently protected.

What marketers need to understand is that *very few search engines matter*.
Recent traffic data suggests that (at least for US traffic) only ~4 spiders
matter. As of spring, SE's ranked by reach, were: Yahoo, MSN, Google, AOL,
AskJeeves (with just under 10%). Every other "search engine" had
considerably less. Yahoo gets its spidered results from Google. MSN and AOL
get their spidered results from Inktomi. AskJeeves gets theirs from Teoma
(themselves). This means that three spiders matter. I would count Alexa in
because they're part of Amazon and you can't ever count Amazon out. That
makes four spiders. I wouldn't lose any sleep over whether DogPile or could find my protected content. The footprints of these spiders
are well known.

Still, the site map option is the cleanest, because no one will call you a
spammer for using it. This other stuff is very questionable, and Google has
zero tolerance for questionable tactics.

Alexis D. Gutzman, Managing Editor
MarketingSherpa's Knowledge Store <- SEO Guide now on sale!

Received on Thu Jul 25 2002 - 11:59:20 CDT


With an archive of more than 14,000 postings, since 1996 the Online Advertising Discussion List has been the Internet's leading forum focused on professional discussion of online advertising and online media buying and selling strategies, results, studies, tools, and media coverage. If you wish to join the discussion list, please use this link to sign up on the home page of the Online Advertising Discussion List.


Online Advertising Industry Leaders:

Local SEO with Video
Houston SEO
Austin Web Design

Add your company...

Local SEO with Video


Online Advertising Discussion List Archives: 2003 - Present
Online Advertising Discussion List Archives: 2001 - 2002
Online Advertising Discussion List Archives: 1999 - 2000
Online Advertising Discussion List Archives: 1996 - 1998

Online Advertising Home | Guidelines | Conferences | Testimonials | Contact Us | Sponsorship | Resources
Site Access and Use Policy | Privacy Policy

2323 Clear Lake City Blvd., Suite 180-139, Houston, TX 77062-8120
Phone: 281-480-6300
Copyright 1996-2007 The Online Advertising Discussion List, a division of ADASTRO Incorporated.
All Rights Reserved.