Welcome to ftp.nluug.nl Current directory: /ftp/os/Linux/distr/salix/i486/extra-14.2/source/perl/perl-www-robotrules/ |
|
Contents of README:This module parses /robots.txt files as specified in "A Standard for Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html> Webmasters can use the /robots.txt file to forbid conforming robots from accessing parts of their web site. The parsed files are kept in a WWW::RobotRules object, and this object provides methods to check if access to a given URL is prohibited. The same WWW::RobotRules object can be used for one or more parsed /robots.txt files on any number of hosts. |
Name Last modified Size
Parent Directory - README 24-Aug-2016 02:43 488 WWW-RobotRules-6.02.tar.gz 24-Aug-2016 02:43 8.8K perl-www-robotrules.SlackBuild 26-Nov-2013 10:57 2.8K perl-www-robotrules.info 24-Aug-2016 02:43 315 slack-desc 24-Aug-2016 02:43 1.0K
NLUUG - Open Systems. Open Standards
Become a member
and get discounts on conferences and more, see the NLUUG website!