Creating A WordPress Robots.txt To Improve SEO
I tried to enter uncharted waters today as I’ve decided to create a robots.txt file, which I haven’t done before. Daniel explains best why this should be a sensible move:
The robots.txt file is used to instruct search engine robots about what pages on your website should be crawled and consequently indexed. Most websites have files and folders that are not relevant for search engines (like images or admin files) therefore creating a robots.txt file can actually improve your website indexation.
The problem I’m having, is I can’t find a definitive source that gives clear instructions on what should be included in a robots.txt so I thought I’d throw my problem out to the Connected Internet ‘team’ to see what should and shouldn’t be included. Here’s what I have so far, which is based mainly on this guide:
User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/
What else do you think I should include or exclude?
Bookmark & Share
Related Posts
- Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days
- Free File Conversion
- Help Me Increase My Feed Subscribers
- Guide To Reducing WordPress Trackback Spam And Comment Spam
- How To Add Related Posts To Your Feed
- Eliminate Wordpress Comment Spam Without A Plugin!
- Permalink Migration Plugin - Great Plugin For SEO
Pingback by Durch robots.txt habe ich die Besucher in 4 Tagen um 16% gesteigert | Connected Internet on 21 April 2007:
[...] ich bereits vor einem Monat in meinem Blog nachfragte wie ich am Besten die robots.txt Datei gestalten sollte um meinen Traffic [...]
Comment by Zath on 12 April 2007:
Another area that I’ve never looked into before, I’ll have to read into this and enlightenment myself, but the idea seems like a sound one to me.
I’m guessing it’s best to start with the defaults shown in the linked file on Daily Blog Tips and go from there adding misc pages I have such as pages full of bookmarks etc.
Cheers!
Comment by Daze on 12 April 2007:
Good idea to disallow the bots from viewing the RSS feeds. My feeds usually always appear above the actual content in the SERPs which is good, but not so great for humans! Thanks, I’ll give this a go
Pingback by Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days | Connected Internet on 12 April 2007:
[...] I asked a month ago for advice on adding a robots.txt file to my WordPress blog to improve my search engine traffic, I didn’t actually get around to it until my blogging [...]
Comment by Everton on 6 April 2007:
Because you don’t want the same posts appearing twice and Google thinking it’s duplicate content, or junk appearing in results as Google will then downgrade your real pages
Comment by hannes on 6 April 2007:
if you want your site indexed, why should you exclude something?
Comment by Mr. Apache on 26 March 2007:
There is a better article about this on the askapache blog:
WordPress robots.txt optimized for SEO
Comment by IndoDX on 26 March 2007:
Why Disallow all extension? such as .PHP, .JS? can I get explain more about it?
Thanks
Pingback by Creating the ultimate WordPress robots.txt file | Twenty Steps on 26 March 2007:
[...] the subject has come up again a couple of times over the last few days at both Wolf-Howl and Connected Internet and pricked my conscience so I thought it was time to revisit [...]
Comment by Mike on 26 March 2007:
One of the reasons why you’d want to use a robots.txt file is to prevent Google from indexing content twice and potentially marking it as duplicate content. If it’s listed in monthly archives, category folders and on your front page, for example, you might end up with pages in the supplemental index.