Spambot Beware

Luring in Spambots

This page gives some warnings about using the scripts from this site, and details some ways to lure spambots in, while keeping regular users away, from the special "anti-spambot" sections.

Topics:


Always use robots.txt

The idea is to catch spambots, which generally don't use robots.txt. You don't want to hurt any friendly robots in the process. Of course, some "friendly-yet-dumb" robots may be caught if they do not read robots.txt, but it is such a widely recognized standard that it's their fault, not yours. Remember, if a spambot does follow the rules in you robots.txt, you still win because it will not be able to access your email addresses, because you have them all stored there.

Using cgi directories

For maximum ability to nail spambots, consider using a cgi directory that is not names "cgi-bin". Some spambots will ignore things in a cgi-bin directory automatically. While this is good if you just want to hide email addresses, to ensure that it visits the traps you have set for it, use a non-standard, innocent sounding name. And, of course, be sure to add the new name to your robots.txt file!

Keep humans away

You also don't want casual viewers wandering through your spamtrap sections. There are two things you should do to ensure this happens:

  1. Hide the entrance
  2. Warn the user

hide the entrance

To hide the entrance, make the link leading to something boring or redundant, which is not likely to be clicked upon by a user browsing the page. You an also make the text of the link very small and unnoticeable, such as the period at the end of the first sentence in this paragraph. :) You can also create an empty link, like this:

<a href="Spambot/notwelcome.html"></A>

(There is an empty link right after the word "create" above. Did you see it? A spambot will. <g>)

Another way is with broken images. Notice how the use of the border=0 hides it very well from some browsers:

[broken image] <a href="#TOP"><img border="0" src="Images/notfound.gif"></a>

Now notice how using the WIDTH and HEIGHT attributes for a perfectly valid image renders it near invisible. (I've left the values as "2" for greater visibility, but feel free to drop it to 1. The image is to the left of the html code below:

o <a href="#TOP"><img height="2" width="2" border="0" src="Images/at.gif"></a>

Warn the users

Warning the users is also important. Let them know, in big bold letters, that they are entering a dangerous area. Tell them it may crash their browser (some of the programs here may). Explain what the area is, if you want, and make a really obvious "fake" link off of the page, while using some of the hidden tricks above to point to the real location.


Watch your resources

Keep an eye on the cgi tricks that you are using - some of them have the potential to suck up a lot of CPU and/or bandwidth, depending on how the spambot acts. Some spambots spawn other processes for new links, so keep that in mind when installing these. You may want to add some extra "sleep" commands into the scripts to slow them down a little.


Spambot Beware: Main page <> Detection <> Avoidance <> Harassment <> Glossary

Written by Greg Sabino Mullane (greg "at" turnstep.com). Last update March 30, 2003.

Valid XHTML 1.0!