Re: Tracking and Looksmart
"Mark Bishop" <markb_at_aala.com> wrote:
> Crawlers are smart enough not to jump into a /cgi-bin directory because of
> the possibility of running into some sort of linking database-a mind field
> to a search engine.
But they're not smart enough to avoid scripts and CGI that are spoofed as
static pages by virtue of having an .htm[l] extension.
> In addition, some crawlers have difficulty in reading
> specific symbols like: ?, &, %, + and $.
> When you use dynamic content that's created on the fly, there's really not
> much an engine can index.
I disagree, based on personal experience building, maintaing and optimizing
sites with dynamic content. A search engine can choose to index all pages,
even pages that require a form POST if that was desirable. If they chose
not to, it's a policy issue, not a technical issue. But if you mean that
many search engines won't *currently* index pages that appear_to_be dynamic,
I'll concur. And like I mentioned in my post on the 18th, it's trivial with
most scripting langauges and webservers to generate dynamic pages that
appear to be static because the end user sees an .htm[l] page without query
strings (the key/value pairs on the right hand side of the question mark in
> However, there are methods that can help you to
> get listed and perhaps obtain a deeper crawl.
> If your only problem is getting past the form page, definitely include as
> that might distract the crawler/spider on this important main page.
> Create a separate "crawler" HTML page that lists your important links on
> it. I often will achieve this by creating by adding a single clear pixel
> with a header tag pointing to my crawler web page. Here's an example:
> <h5><a href="crawler_mywebsite.html"><img src="clearpixel1.gif" height="1"
> width="1" border="0" alt="Description Here"></h5></a>
It's been my understanding that some spiders would consider such HTML as
hidden text and therefore would not index the linked page. Anyone have
thoughts on that? Info. on which search engines this technique works with
and which it doesn not? I do admire the technique since all, but the most
savvy or investigative human visitors won't see the linked page.
President, Befriend Internet Services LLC
Received on Wed Jul 24 2002 - 09:58:57 CDT
HOW TO JOIN THE ONLINE ADVERTISING DISCUSSION LIST
With an archive of more than 14,000 postings, since 1996 the
Online Advertising Discussion List has been the Internet's leading forum focused on professional discussion
of online advertising and online media buying and selling strategies, results, studies, tools, and media
coverage. If you wish to join the discussion list, please use this link to sign up on the home page of the Online Advertising Discussion List.|
Online Advertising Industry Leaders:
Local SEO with Video
Austin Web Design
Add your company...