Creating A WordPress Robots.txt To Improve SEO
I tried to enter uncharted waters today as I’ve decided to create a robots.txt file, which I haven’t done before. Daniel explains best why this should be a sensible move:
The robots.txt file is used to instruct search engine robots about what pages on your website should be crawled and consequently indexed. Most websites have files and folders that are not relevant for search engines (like images or admin files) therefore creating a robots.txt file can actually improve your website indexation.
The problem I’m having, is I can’t find a definitive source that gives clear instructions on what should be included in a robots.txt so I thought I’d throw my problem out to the Connected Internet ‘team’ to see what should and shouldn’t be included. Here’s what I have so far, which is based mainly on this guide:
User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/
What else do you think I should include or exclude?
Bookmark & Share
Related Posts
- Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days
- Free File Conversion
- Help Me Increase My Feed Subscribers
- Guide To Reducing WordPress Trackback Spam And Comment Spam
- How To Add Related Posts To Your Feed
- Eliminate Wordpress Comment Spam Without A Plugin!
- Permalink Migration Plugin - Great Plugin For SEO
Comment by Paula Mooney on 9 October 2007:
Thank you so much for this post.
I found out I had my robots.txt file in the wrong place anyway!
Paula
Pingback by Simple SEO Friendly Robots.txt File for Wordpress | Double Black Design on 6 October 2007:
[...] format for a robots.txt file for a site utilizing Wordpress. I have found various simple and complex file formats but I am yet to determine which format is really the [...]
Pingback by 10 Great WordPress SEO Articles — www.aruntheace.com on 12 August 2007:
[...] Creating a WordPress Robots.txt to Improve SEO Robots.txt file is used to tell the search engine bots how to crawl your blog. This article helps you to create a Robots.txt file for WordPress blogs. [...]
Comment by Ask Apache on 9 August 2007:
Hey Everton just stopped by to let you know I created a simpler and updated robots.txt file for WordPress, and I used a lot of the recommendations from your visitor comments.
Pingback by Tuto de l’été n°3: créez un fichier robots.txt pour votre blog Wordpress on 7 August 2007:
[...] Creating A WordPress Robots.txt To Improve SEO de ConnectedInternet, [...]
Comment by Ramil on 6 August 2007:
Mike has a good point about duplicate contents
Pingback by Wordpress Kullanıcıları İçin robot.txt dosyası | Bilginciler.com on 24 July 2007:
[...] size bi rorobt.txt örneği [Creating A WordPress Robots.txt To Improve SEO | Connected Internet ] ama ben pek sevmedim yani nie [feeed - page’ı engellemşki] neyse ben paylaşımde isteyen [...]
Pingback by Silkenhut’s World » Blog Archive » Add a robots.txt file to your blog on 16 July 2007:
[...] Creating a wordpress robots.txt to improve seo [...]
Pingback by How To Validate Your Robots.txt File · Make Money Online With CMB on 13 June 2007:
[...] needs to be located at http://www.yourdomain.com/robots.txt for Search Engines to find it. To see if your robots.txt file is online simple enter the above URL into your favorite browser - the text file should display [...]
Pingback by Wordpress Kullanıcıları İçin robot.txt dosyası | Wordpress, Seo, Webmaster Makaleleri on 18 May 2007:
[...] size bi rorobt.txt örneÄŸi [Creating A WordPress Robots.txt To Improve SEO | Connected Internet ] ama ben pek sevmedim yani nie [feeed - page’ı engellemÅŸki] neyse ben paylaşımde isteyen [...]