I tried to enter uncharted waters today as I’ve decided to create a robots.txt file, which I haven’t done before. Daniel explains best why this should be a sensible move:
The robots.txt file is used to instruct search engine robots about what pages on your website should be crawled and consequently indexed. Most websites have files and folders that are not relevant for search engines (like images or admin files) therefore creating a robots.txt file can actually improve your website indexation.
The problem I’m having, is I can’t find a definitive source that gives clear instructions on what should be included in a robots.txt so I thought I’d throw my problem out to the Connected Internet ‘team’ to see what should and shouldn’t be included. Here’s what I have so far, which is based mainly on this guide:
User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/
What else do you think I should include or exclude?
