Re: Tracking and Looksmart

From: Steve Werby <steve-lists_at_befriend.com>
Date: Wed 24 Jul 2002 09:58:28 -0500

"Alexis D. Gutzman" <alexisg_at_marketingsherpa.com> wrote:
> Rather than trying to fool the search engines, why not do what they
> recommend, and put a site map link on the home page, which the spider will
> find and crawl.

A good idea, if it's feasible.

> This will put all or most of your content within two clicks
> of the home page, and make all of it readily accessible to the search
> engine. As long as you make human navigation clear, most visitors won't
> click on the site map. Putting the site map link below the fold will also
> help, since a site we recently profiled found that ~90% of visitors never
> even scrolled!

And I always thought the main purpose of the site map was so humans could
find out what's on the site. Live and learn. LOL.

> If you need to be absolutely sure that only spiders make it
> to the protected pages without getting the form, you can use server-side
> detection to identify the footprint of the spider (there aren't that many
> that really matter) and let the server show these agents the content
behind
> the barrier without seeing the form.

I think your reply and my earlier reply to Jill crossed paths, but I want to
reiterate that this isn't foolproof. If you know what user agents most
search engines regularly use and you know how to set your user agent, then
you can trick sites that don't know better into thinking you are a speciif
search engine. In fact, that's a common technique used to view content,
including meta tags, which competitors publish for specific search engines.
Plus, many search engines have caught onto the specialized content/tag game
and are now indexing, at least in part, with different user agents and
sometimes different IP ranges to determine if sites are using this
technique. If you combine user agent and IPs it'll be a better solution,
though coming by the info. required to do that will probably be difficult.

--
Steve Werby
President, Befriend Internet Services LLC
http://www.befriend.com/






Received on Wed Jul 24 2002 - 09:58:28 CDT