Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days


I finally got around to adding a Robots.txt file on Sunday and already my Google Traffic has gone up by 16% in only 4 days. I’m amazed that such a small change as adding a robots.txt file has had such an impact.

Although I asked a month ago for advice on adding a robots.txt file to my WordPress blog to improve my search engine traffic, I didn’t actually get around to it until my blogging buddy in arms Martin emailed me to tell me that I had 26,000 supplemental links which wasn’t good, and advised I added a robots.txt file to my WordPress installation.

Well, after only 4 days my supplemental links have fallen by 12,600 and my Google search engine traffic has gone up by 16% over the same period. The supplemental index is basically where Google stores all the pages that rank lowely, which are displayed after Google has tried to match pages from its main index to search terms. The more pages you can get into the main index the greater your changes of getting traffic from Google.

I’m 100% convinced that the increase in Google search engine traffic is directly related to my adding a robots.txt file. Unlike the increase in my Alexa rank where I can’t say for certain that it’s a result of my Alexa Rank experiment, I’m confident that the robots.txt file has caused the increase in my search engine traffic, as I pay close attention to the numbers everyday and I know all the trends. What I believe has happened is that a significant number of my pages have been moved from the supplemental index to the main index, which has increased the number of times my site appears in results and how high my results appear.

It’d be interesting to hear if anyone else has had a similar experience from adding a robots.txt file. If you haven’t added a robots.txt file to your site then the one I’m using is below. To check te number of supplemental links that your site has just enter ‘site:www.YOURDOMAIN.COM -view ***‘ into google and note the number of results that are returned.

User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/

Related Posts

Filed Under: BloggingTools & Tips

Tags:

Bookmark & Share

About the Author: Everton is based in London and has worked in the internet and mobile space for over ten years now, and before that worked in corporate strategy and consulting. He has a degree in Economics from Cambridge University, and currently runs the Portal and online operations for one of the largest ISPs in the UK. He also writes for Windows 8 News, Windows 8 News and One Tip A Day.

RSSComments (44)

Leave a Reply | Trackback URL

  1. Ask Apache says:

    Bravo! I just took a look at your robots.txt file and I would recommend taking a look at the updated wordpress robots.txt file on askapache.

  2. About three weeks ago, I finally got around to uploading the robots.txt file for http://www.freelancersatlarge.com (it had been way down on the list of prorities).

    In three weeks, the number of links in Google’s index increased by 5 times!

  3. hello friend, i rode your article as a robots.txt and you say that by robots file increase the traffic then can u clearly told me how to possible…

  4. patrick says:

    Do you think this works on a none word press site?

Trackbacks/Pingbacks

  1. Revisiting robots.txt | Twenty Steps
  2. Increase Traffic With Robots.txt | WordPress SEO
  3. clickden.net » Blog Archive » BTTB DL Links
  4. Ignore Traffic for More Traffics! | Blogging on Tech, Web 2.0 and Money Making Tips
  5. Tuto de l’été n°3: créez un fichier robots.txt pour votre blog Wordpress
  6. Using robots.txt for SEO » Omnistar Etools
  7. Optimize Your Personal Wordpress Blog | drew tarvin blog

Subscribe without commenting

Leave a Reply

CommentLuv Enabled
Comment Policy: Any comments are permitted only because the site owner is letting you post, and any comments could be removed for any reason at the absolute discretion of the site owner.