Re: Tracking and Looksmart

From: Steve Werby <>
Date: Wed 24 Jul 2002 09:58:28 -0500

"Alexis D. Gutzman" <> wrote:
> Rather than trying to fool the search engines, why not do what they
> recommend, and put a site map link on the home page, which the spider will
> find and crawl.

A good idea, if it's feasible.

> This will put all or most of your content within two clicks
> of the home page, and make all of it readily accessible to the search
> engine. As long as you make human navigation clear, most visitors won't
> click on the site map. Putting the site map link below the fold will also
> help, since a site we recently profiled found that ~90% of visitors never
> even scrolled!

And I always thought the main purpose of the site map was so humans could
find out what's on the site. Live and learn. LOL.

> If you need to be absolutely sure that only spiders make it
> to the protected pages without getting the form, you can use server-side
> detection to identify the footprint of the spider (there aren't that many
> that really matter) and let the server show these agents the content
> the barrier without seeing the form.

I think your reply and my earlier reply to Jill crossed paths, but I want to
reiterate that this isn't foolproof. If you know what user agents most
search engines regularly use and you know how to set your user agent, then
you can trick sites that don't know better into thinking you are a speciif
search engine. In fact, that's a common technique used to view content,
including meta tags, which competitors publish for specific search engines.
Plus, many search engines have caught onto the specialized content/tag game
and are now indexing, at least in part, with different user agents and
sometimes different IP ranges to determine if sites are using this
technique. If you combine user agent and IPs it'll be a better solution,
though coming by the info. required to do that will probably be difficult.

Steve Werby
President, Befriend Internet Services LLC

Received on Wed Jul 24 2002 - 09:58:28 CDT


With an archive of more than 14,000 postings, since 1996 the Online Advertising Discussion List has been the Internet's leading forum focused on professional discussion of online advertising and online media buying and selling strategies, results, studies, tools, and media coverage. If you wish to join the discussion list, please use this link to sign up on the home page of the Online Advertising Discussion List.


Online Advertising Industry Leaders:

Local SEO with Video
Houston SEO
Austin Web Design

Add your company...

Local SEO with Video


Online Advertising Discussion List Archives: 2003 - Present
Online Advertising Discussion List Archives: 2001 - 2002
Online Advertising Discussion List Archives: 1999 - 2000
Online Advertising Discussion List Archives: 1996 - 1998

Online Advertising Home | Guidelines | Conferences | Testimonials | Contact Us | Sponsorship | Resources
Site Access and Use Policy | Privacy Policy

2323 Clear Lake City Blvd., Suite 180-139, Houston, TX 77062-8120
Phone: 281-480-6300
Copyright 1996-2007 The Online Advertising Discussion List, a division of ADASTRO Incorporated.
All Rights Reserved.