Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days
I finally got around to adding a Robots.txt file on Sunday and already my Google Traffic has gone up by 16% in only 4 days. I’m amazed that such a small change as adding a robots.txt file has had such an impact.
Although I asked a month ago for advice on adding a robots.txt file to my WordPress blog to improve my search engine traffic, I didn’t actually get around to it until my blogging buddy in arms Martin emailed me to tell me that I had 26,000 supplemental links which wasn’t good, and advised I added a robots.txt file to my WordPress installation.
Well, after only 4 days my supplemental links have fallen by 12,600 and my Google search engine traffic has gone up by 16% over the same period. The supplemental index is basically where Google stores all the pages that rank lowely, which are displayed after Google has tried to match pages from its main index to search terms. The more pages you can get into the main index the greater your changes of getting traffic from Google.
I’m 100% convinced that the increase in Google search engine traffic is directly related to my adding a robots.txt file. Unlike the increase in my Alexa rank where I can’t say for certain that it’s a result of my Alexa Rank experiment, I’m confident that the robots.txt file has caused the increase in my search engine traffic, as I pay close attention to the numbers everyday and I know all the trends. What I believe has happened is that a significant number of my pages have been moved from the supplemental index to the main index, which has increased the number of times my site appears in results and how high my results appear.
It’d be interesting to hear if anyone else has had a similar experience from adding a robots.txt file. If you haven’t added a robots.txt file to your site then the one I’m using is below. To check te number of supplemental links that your site has just enter ‘site:www.YOURDOMAIN.COM -view ***‘ into google and note the number of results that are returned.
User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/
Bookmark & Share
Pingback by Using robots.txt for SEO » Omnistar Etools on 6 November 2007:
[...] a robots.txt file will increase your ranking in the search engines. It’s possible, and some people think they have evidence for it. But the answer is not really clear, as it isn’t the kind of thing that’s important to [...]
Comment by FreelancersAtLarge.com on 4 November 2007:
About three weeks ago, I finally got around to uploading the robots.txt file for http://www.freelancersatlarge.com (it had been way down on the list of prorities).
In three weeks, the number of links in Google’s index increased by 5 times!
Comment by Ask Apache on 9 August 2007:
Bravo! I just took a look at your robots.txt file and I would recommend taking a look at the updated wordpress robots.txt file on askapache.
Pingback by Tuto de l’été n°3: créez un fichier robots.txt pour votre blog Wordpress on 7 August 2007:
[...] Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days, aussi de ConnectedInterne…. [...]
Comment by Everton on 29 July 2007:
I think you need:
User-agent: Googlebot
Disallow: /*/welcome/$
User-agent: *
Disallow: /news/welcome/
Hoepfully someone else will confirm
Comment by Max on 27 July 2007:
If I want to block a subdirectory only. e.g. http://www.site.com/news/welcome
What command should I sue:
1. Disallow: /news/welcome/
2. Disallow: /*/welcome/
3. Disallow: /welcome
I know number 1 should be 100% works. But sometimes, I don’t want to type the exact path.
As for number 2, I think it works for google and yahoo bot only. I only these 2 which accept “*”.
As for number 3, I don’t know whether it works or not. As for as I understand, with or without “/” make a big differences. With “/”, you are saying to that directory only. Without “/”, you are saying anything start from /welcome. Please correct me if I am wrong.
So the subdirectory news/welcome also start from /welcome, so I can’t sure whether Disallow: /welcome will block this subdirectory or not.
Pingback by Ignore Traffic for More Traffics! | Blogging on Tech, Web 2.0 and Money Making Tips on 10 July 2007:
[...] Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days [...]
Pingback by clickden.net » Blog Archive » BTTB DL Links on 3 July 2007:
[...] Finally, get a robots.txt file set up for the search engine crawlers. I use one I found at Connected internet Blog. Google and other crawlers expect to find a robots.txt in your root directory. Many of the error [...]