All bots banned

Update: Harsh ban removed

For those search engine bots and whatever else obeys robots.txt, I’ve disallowed access to my whole server. I’ll do this for a month, to clear the cache, then re-enable access (except for any image directories). I’ll be better with this stuff in the future.

Why am I doing this? Maybe it’ll clear things up in my logs and whatnot, especially pages I probably don’t want accessed anymore. Besides, I think the majority of my traffic comes directly from going to my blog. Forums are pretty much dead. The old site is there because of some pages that are being accessed, but I’ll probably just redirect all to a single page stating that it was taken offline.

So again, all bots are banned. I’ll rethink and redo everything a month from now.

If this is all jumbled and confusing, it’s because I’m kind of doing all this off the top of my head. If you want to take a look at my tiny robots.txt file, go ahead. Do I need to do anything different for subdomains? Do they need their own robots.txt file? Or will that single slash cover everything under the sun?

Published by

Bryan Villarin

Bryan is a Community Guardian at Automattic. He's also a photographer, card magician, and cat whisperer. (Thanks to my friend and colleague Steve Blythe for the sweet photo!)

6 thoughts on “All bots banned”

  1. (sorry for typing faster than I think)

    And, that single slash will cover everything under each domain, so your robots.txt is kind of redundant. All you need is this (one robots.txt file for each domain/subdomain):

    User-agent: *
    Disallow: /

    Like

  2. I can’t help but think this will have ill effects.

    Anyways. What is wrong with people coming in from pages that do not exist? I suppose it matters much less in my case, because all 404 pages are custom. I even create custom ones for certain subdomains.

    Like

  3. @James: all the changes you’ve suggested have been implemented. I installed Bad Behavior right afterwards, too. Do you check your logs at all?

    @Anth: What do you think will go wrong?

    Like

  4. I check my BadBehavior logs every day and have them set to only store entries that are up to 2 days old. Some days, the log only has 20 entries in it. And on days like yesterday, the log may be filled with over 600. Sometimes you’ll see things that look like false positives, but if you send the developer the MySQL data for that entry, he’ll look into it for you. So far, everything that I thought was a false positive, turned out to be a bad bot.

    Like

Talk to me, Goose

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s