Creating A WordPress Robots.txt To Improve SEO
Everton | Mar 25, 2007 | Comments
I tried to enter uncharted waters today as I’ve decided to create a robots.txt file, which I haven’t done before. Daniel explains best why this should be a sensible move:
The robots.txt file is used to instruct search engine robots about what pages on your website should be crawled and consequently indexed. Most websites have files and folders that are not relevant for search engines (like images or admin files) therefore creating a robots.txt file can actually improve your website indexation.
The problem I’m having, is I can’t find a definitive source that gives clear instructions on what should be included in a robots.txt so I thought I’d throw my problem out to the Connected Internet ‘team’ to see what should and shouldn’t be included. Here’s what I have so far, which is based mainly on this guide:
User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/
What else do you think I should include or exclude?
Read Related Posts
Filed Under: Blogging • Tools & Tips
About the Author: Everton is based in London and has worked in the internet and mobile space for over ten years now, and before that worked in corporate strategy and consulting. He has a degree in Economics from Cambridge University.He also writes for Windows 7 News, Windows 8 News and One Tip A Day.
- Daniel
- Lisa
- Leftblank
- Everton
- Audio Books Fan
- SoccerSpeech
- bill
- SoKoOLz
- Thilak
- SoccerSpeech
Webmastery 101: How To Cut Fat And Build Muscle
Get Free MP3s at Sideload



