Re: Tracking and Looksmart

From: Steve Werby <>
Date: Wed 24 Jul 2002 09:58:57 -0500

"Mark Bishop" <> wrote:
> Crawlers are smart enough not to jump into a /cgi-bin directory because of
> the possibility of running into some sort of linking database-a mind field
> to a search engine.

But they're not smart enough to avoid scripts and CGI that are spoofed as
static pages by virtue of having an .htm[l] extension.

> In addition, some crawlers have difficulty in reading
> specific symbols like: ?, &, %, + and $.
> When you use dynamic content that's created on the fly, there's really not
> much an engine can index.

I disagree, based on personal experience building, maintaing and optimizing
sites with dynamic content. A search engine can choose to index all pages,
even pages that require a form POST if that was desirable. If they chose
not to, it's a policy issue, not a technical issue. But if you mean that
many search engines won't *currently* index pages that appear_to_be dynamic,
I'll concur. And like I mentioned in my post on the 18th, it's trivial with
most scripting langauges and webservers to generate dynamic pages that
appear to be static because the end user sees an .htm[l] page without query
strings (the key/value pairs on the right hand side of the question mark in
a URL).

> However, there are methods that can help you to
> get listed and perhaps obtain a deeper crawl.
> If your only problem is getting past the form page, definitely include as
> many links as possible. Avoid frames, Flash, Javascript, or anything else
> that might distract the crawler/spider on this important main page.
> Create a separate "crawler" HTML page that lists your important links on
> it. I often will achieve this by creating by adding a single clear pixel
> with a header tag pointing to my crawler web page. Here's an example:
> <h5><a href="crawler_mywebsite.html"><img src="clearpixel1.gif" height="1"
> width="1" border="0" alt="Description Here"></h5></a>

It's been my understanding that some spiders would consider such HTML as
hidden text and therefore would not index the linked page. Anyone have
thoughts on that? Info. on which search engines this technique works with
and which it doesn not? I do admire the technique since all, but the most
savvy or investigative human visitors won't see the linked page.

Steve Werby
President, Befriend Internet Services LLC

Received on Wed Jul 24 2002 - 09:58:57 CDT


With an archive of more than 14,000 postings, since 1996 the Online Advertising Discussion List has been the Internet's leading forum focused on professional discussion of online advertising and online media buying and selling strategies, results, studies, tools, and media coverage. If you wish to join the discussion list, please use this link to sign up on the home page of the Online Advertising Discussion List.


Online Advertising Industry Leaders:

Local SEO with Video
Houston SEO
Austin Web Design

Add your company...

Local SEO with Video


Online Advertising Discussion List Archives: 2003 - Present
Online Advertising Discussion List Archives: 2001 - 2002
Online Advertising Discussion List Archives: 1999 - 2000
Online Advertising Discussion List Archives: 1996 - 1998

Online Advertising Home | Guidelines | Conferences | Testimonials | Contact Us | Sponsorship | Resources
Site Access and Use Policy | Privacy Policy

2323 Clear Lake City Blvd., Suite 180-139, Houston, TX 77062-8120
Phone: 281-480-6300
Copyright 1996-2007 The Online Advertising Discussion List, a division of ADASTRO Incorporated.
All Rights Reserved.