Creating A WordPress Robots.txt To Improve SEO


I tried to enter uncharted waters today as I’ve decided to create a robots.txt file, which I haven’t done before. Daniel explains best why this should be a sensible move:

The robots.txt file is used to instruct search engine robots about what pages on your website should be crawled and consequently indexed. Most websites have files and folders that are not relevant for search engines (like images or admin files) therefore creating a robots.txt file can actually improve your website indexation.

The problem I’m having, is I can’t find a definitive source that gives clear instructions on what should be included in a robots.txt so I thought I’d throw my problem out to the Connected Internet ‘team’ to see what should and shouldn’t be included. Here’s what I have so far, which is based mainly on this guide:


User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/

What else do you think I should include or exclude?

Related Posts

Filed Under: BloggingTools & Tips

Tags:

Bookmark & Share

About the Author: Everton is based in London and has worked in the internet and mobile space for over ten years now, and before that worked in corporate strategy and consulting. He has a degree in Economics from Cambridge University, and currently runs the Portal and online operations for one of the largest ISPs in the UK. He also writes for Windows 8 News, Windows 8 News and One Tip A Day.

RSSComments (30)

Leave a Reply | Trackback URL

  1. Everton says:

    Thx Sue for reminding me I need to check my file

  2. AskApache says:

    Have you read the new robots.txt recently updated article yet? and check out askapache.com/robots.txt

  3. hi, thanks for posting this.
    i use it all unkess :
    Disallow: /page/
    Disallow: /date/
    Disallow: /comments/

  4. stacey says:

    Hi Everett,

    I installed the KB robots.txt plugin. This what I entered in the robots.txt plugin window.
    User-agent: *

    Disallow:
    But when I do I do http://www.babygeartoday.com/robots.txt this is what I get, and in the plugin when it says check the the robots.txt file after I submit I get this also:
    # BEGIN XML-SITEMAP-PLUGIN
    Sitemap: http://www.babygeartoday.com/sitemap.xml.gz
    # END XML-SITEMAP-PLUGIN

    So uninstalled the google sitemap and analytics, and still the same thing. I was wondering could you help me solve this problem.

    Thanks,
    Stacey

  5. stacey says:

    I fixed the robots.txt. I wasnt putting it in the root directory.

  6. AskApache says:

    I finally got my site indexed near perfectly thanks to much reading and viewing my access logs.. Based mostly on this wordpress robots.txt example

  7. LiveJasmin says:

    should this be ok ?

    User-agent: *
    Allow: /wp-content/uploads/
    Disallow: /wp-content/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-
    Disallow: /feed/
    Disallow: /trackback/
    Disallow: /cgi-bin/

    User-agent: Googlebot-Image
    Disallow:
    Allow: /*

  8. What about to block search results?

  9. I think the following site describes perfectly the seo adjustments for a wordpress blog’s robots.txt:
    http://codex.wordpress.org/Search_Engine_Optimization_for_Wordpress

Trackbacks/Pingbacks

  1. Wordpress Kullanıcıları İçin robot.txt dosyası | Rasim Coşkun | Web Tasarım | Web Design
  2. Poor SERPS and Landing pages » Eigenmatrix.org
  3. Add Robots.txt to get traffic increase | Sha Money Maker

Subscribe without commenting

Leave a Reply

CommentLuv Enabled
Comment Policy: Any comments are permitted only because the site owner is letting you post, and any comments could be removed for any reason at the absolute discretion of the site owner.