Creating A WordPress Robots.txt To Improve SEO
Everton | Mar 25, 2007 | Comments 30
I tried to enter uncharted waters today as I’ve decided to create a robots.txt file, which I haven’t done before. Daniel explains best why this should be a sensible move:
The robots.txt file is used to instruct search engine robots about what pages on your website should be crawled and consequently indexed. Most websites have files and folders that are not relevant for search engines (like images or admin files) therefore creating a robots.txt file can actually improve your website indexation.
The problem I’m having, is I can’t find a definitive source that gives clear instructions on what should be included in a robots.txt so I thought I’d throw my problem out to the Connected Internet ‘team’ to see what should and shouldn’t be included. Here’s what I have so far, which is based mainly on this guide:
User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/
What else do you think I should include or exclude?
Related Posts
- Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days
- Help Me Improve Connected Internet
- Eliminate Wordpress Comment Spam Without A Plugin!
- Guide To Reducing WordPress Trackback Spam And Comment Spam
- Webmastery 101: How To Cut Fat And Build Muscle
- The Dos And Don’ts For Driving Traffic To A New Website
- Get Free MP3s at Sideload
Filed Under: Blogging • Tools & Tips
Bookmark & Share
Comments (30)
Trackbacks/Pingbacks
- Wordpress Kullanıcıları İçin robot.txt dosyası | Rasim Coşkun | Web Tasarım | Web Design
- Poor SERPS and Landing pages » Eigenmatrix.org
- Add Robots.txt to get traffic increase | Sha Money Maker
FWIW: this tester shows a lot of errors:
http://www.searchenginepromotionhelp.com/m/robots-text-tester/robots-checker.php
Sue
Thx Sue for reminding me I need to check my file
Have you read the new robots.txt recently updated article yet? and check out askapache.com/robots.txt
hi, thanks for posting this.
i use it all unkess :
Disallow: /page/Disallow: /date/
Disallow: /comments/
Hi Everett,
I installed the KB robots.txt plugin. This what I entered in the robots.txt plugin window.
User-agent: *
Disallow:
But when I do I do http://www.babygeartoday.com/robots.txt this is what I get, and in the plugin when it says check the the robots.txt file after I submit I get this also:
# BEGIN XML-SITEMAP-PLUGIN
Sitemap: http://www.babygeartoday.com/sitemap.xml.gz
# END XML-SITEMAP-PLUGIN
So uninstalled the google sitemap and analytics, and still the same thing. I was wondering could you help me solve this problem.
Thanks,
Stacey
I fixed the robots.txt. I wasnt putting it in the root directory.
I finally got my site indexed near perfectly thanks to much reading and viewing my access logs.. Based mostly on this wordpress robots.txt example
should this be ok ?
User-agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /cgi-bin/
User-agent: Googlebot-Image
Disallow:
Allow: /*
What about to block search results?
I think the following site describes perfectly the seo adjustments for a wordpress blog’s robots.txt:
http://codex.wordpress.org/Search_Engine_Optimization_for_Wordpress