Google Please Don’t Crawl This Server

For some reason the Googlebot has found it necessary to crawl a development server of mine, I suspect that one of the users uses Google Chrome which probably snarfs urls browsed.

Google tells us that one way to do this is to return the 410 HTTP  status code , and the way to enforce this in httpd.conf is:

# Tell crawlers to go away   
RewriteEngine On    
RewriteCond %{HTTP_USER_AGENT} (Googlebot|bingbot|Validator|MJ12bot|Baiduspider)    
RewriteRule ^.* - [G]


I have included other crawlers in the list just to make sure.



Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: