If you are visiting this page you probably saw our web crawler, PDFBot, accessing your web site.
We are a building an index of PDF-media on the web. We've put much effort into being a good internet citizen by making our crawler as polite as possible.
We understand your bandwidth and server time are valuable. It is important for us and for our business to be welcomed on the internet as a well-behaved crawler. Although we try very hard to be polite there is always the possibility of a bug. If you notice any unusual activity from PDFBot please report it to: crawler -at- pdfind.com
We want to hear from you and we will respond promptly. Your feedback has helped us to improve PDFBot.
If you have a problem or concern about PDFBot we much prefer to have the chance to address it but if you need to block PDFBot we do respect the robots.txt exclusion list. To block PDFBot from some parts of your web site you can use the following example:
User-agent: PDFBot
Disallow: /upload_dir/
Disallow: /my_stuff/
In this example, /upload_dir/ and /pdf_stuff/ are directories that will be blocked to PDFBot and won't be crawled. Other parts of your web site will still be crawled.
To block PDFBot from your entire web site you can use this:
User-agent: PDFBot
Disallow: /
Please note, our web crawler caches robots.txt files and it can take up to 48 hours before it is re-read.
More information on robots.txt can be found at http://www.robotstxt.org