TWiki . Main . CageyCrawlers TWiki . { Main | Edumacation | TWiki | Test }
Main . { Home | Users | Changes | Search | Go }
Search engines that crawl the Cagey Consumer web site:

search engine robot name robot info page
Alexa ia_archiver
alltheweb fastwebcrawler
Alta Vista Scooter-3.0.FS none
Commission Junction CJNetworkQuality
Digital Integrity DIIbot/1.2 Pompos/1.3
Direct Hit Mozilla/2.0 none
Domanova Jack
Excite ArchitextSpider none Pompos/1.3
GAIS Openbot/3.0
Girafabot girafabot
GoGettem none
google Googlebot/2.1
Links2Go Links2Go Similarity Engine
Lycos Lycos_Spider_(T-Rex) none
NameProtect NPBot
ignores robots.txt
National Directory NationalDirectory-WebSpider/1.3 none
Northern Light Search Gulliver/1.3 none
Openfind Openbot/3.0+
Picsearch psbot/0.1
Planet Internet appie/1.1 none
PolySearch polybot 1.0
Teoma teomaagent none
wisenut ZyBorg/1.0
WebTop MuscatFerret/2.0 none
( tivraSpider/1.0 none
( MSIE 4.01
ignores robots.txt
IP address
( Wget/1.6 none
( none note: uses http/1.1 but sends wrong host name for twiki
( LinkWalker none
( cosmos/0.8 none
? JennyBot/0.1 none
? Bjaaland/0.9

For info on robot exclusion (robots.txt) files, see
Here are some sites which try to track various crawlers:

Topic CageyCrawlers . { Edit | Ref-By | Attach | Diffs | r1.18 | > | r1.17 | > | r1.16 | > | r1.15 | >... }
You must register before editing pages or using other extended features on this TWiki system.
Revision r1.18 - 02 Aug 2003 - 04:16 by EliMantel web search for EliMantel
Privacy Policy
Copyright © 2000-2005 by the contributing authors. All material on this collaboration tool is the property of the contributing authors. Collect email addresses here.
Ideas, requests, problems regarding TWiki? Send feedback.