Creating A WordPress Robots.txt To Improve SEO
I tried to enter uncharted waters today as I’ve decided to create a robots.txt file, which I haven’t done before. Daniel explains best why this should be a sensible move:
The robots.txt file is used to instruct search engine robots about what pages on your website should be crawled and consequently indexed. Most websites have files and folders that are not relevant for search engines (like images or admin files) therefore creating a robots.txt file can actually improve your website indexation.
The problem I’m having, is I can’t find a definitive source that gives clear instructions on what should be included in a robots.txt so I thought I’d throw my problem out to the Connected Internet ‘team’ to see what should and shouldn’t be included. Here’s what I have so far, which is based mainly on this guide:
User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/
What else do you think I should include or exclude?
Tip: Click here to run a free scan for common PC errors
loading...
Category: Blogging, Tools & Tips
Next Post: Increasing Alexa Rank – A Joint Experiment »» Prev Post: Why You Should Think Twice Before Using Translator Plugins »»







Tiger always gives 110 percent. That is why he gave 100 percent to his wife and still had 10 percent left over for his alleged mistress.
I enjoyed reading this, are you on twitter?
Great post. A lot of useful reviews here. I’m sending it to my friends!