Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days

I finally got around to adding a Robots.txt file on Sunday and already my Google Traffic has gone up by 16% in only 4 days. I’m amazed that such a small change as adding a robots.txt file has had such an impact.

Although I asked a month ago for advice on adding a robots.txt file to my WordPress blog to improve my search engine traffic, I didn’t actually get around to it until my blogging buddy in arms Martin emailed me to tell me that I had 26,000 supplemental links which wasn’t good, and advised I added a robots.txt file to my WordPress installation.

Well, after only 4 days my supplemental links have fallen by 12,600 and my Google search engine traffic has gone up by 16% over the same period. The supplemental index is basically where Google stores all the pages that rank lowely, which are displayed after Google has tried to match pages from its main index to search terms. The more pages you can get into the main index the greater your changes of getting traffic from Google.

I’m 100% convinced that the increase in Google search engine traffic is directly related to my adding a robots.txt file. Unlike the increase in my Alexa rank where I can’t say for certain that it’s a result of my Alexa Rank experiment, I’m confident that the robots.txt file has caused the increase in my search engine traffic, as I pay close attention to the numbers everyday and I know all the trends. What I believe has happened is that a significant number of my pages have been moved from the supplemental index to the main index, which has increased the number of times my site appears in results and how high my results appear.

It’d be interesting to hear if anyone else has had a similar experience from adding a robots.txt file. If you haven’t added a robots.txt file to your site then the one I’m using is below. To check te number of supplemental links that your site has just enter ‘site:www.YOURDOMAIN.COM -view ***‘ into google and note the number of results that are returned.

User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/