Comments on: Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/ The latest tech, mobile and gaming news Sun, 21 Mar 2010 06:42:50 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: wp-popular.com » Blog Archive » Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days | Connected Internethttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-6/#comment-175694 wp-popular.com » Blog Archive » Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days | Connected Internet Mon, 08 Mar 2010 18:15:23 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-175694 [...] Go here to see the original: Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days | Connected Internet [...] [...] Go here to see the original: Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days | Connected Internet [...]

]]> By: patrickhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-6/#comment-155490 patrick Thu, 12 Feb 2009 03:37:58 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-155490 Do you think this works on a none word press site? Do you think this works on a none word press site?

]]> By: Webrecsol Optimizerhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-155278 Webrecsol Optimizer Wed, 21 Jan 2009 05:47:11 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-155278 hello friend, i rode your article as a robots.txt and you say that by robots file increase the traffic then can u clearly told me how to possible... hello friend, i rode your article as a robots.txt and you say that by robots file increase the traffic then can u clearly told me how to possible…

]]> By: Optimize Your Personal Wordpress Blog | drew tarvin bloghttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-153688 Optimize Your Personal Wordpress Blog | drew tarvin blog Wed, 10 Sep 2008 00:35:59 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-153688 [...] your Robots.txt file - follow the guide at http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffi... [...] [...] your Robots.txt file – follow the guide at http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffi... [...]

]]> By: Using robots.txt for SEO » Omnistar Etoolshttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-88276 Using robots.txt for SEO » Omnistar Etools Tue, 06 Nov 2007 16:18:52 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-88276 [...] a robots.txt file will increase your ranking in the search engines. It’s possible, and some people think they have evidence for it. But the answer is not really clear, as it isn’t the kind of thing that’s important to [...] [...] a robots.txt file will increase your ranking in the search engines. It’s possible, and some people think they have evidence for it. But the answer is not really clear, as it isn’t the kind of thing that’s important to [...]

]]> By: FreelancersAtLarge.comhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-88019 FreelancersAtLarge.com Sun, 04 Nov 2007 08:32:04 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-88019 About three weeks ago, I finally got around to uploading the robots.txt file for www.freelancersatlarge.com (it had been way down on the list of prorities).In three weeks, the number of links in Google's index increased by 5 times! About three weeks ago, I finally got around to uploading the robots.txt file for http://www.freelancersatlarge.com (it had been way down on the list of prorities).

In three weeks, the number of links in Google’s index increased by 5 times!

]]> By: Ask Apachehttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-71375 Ask Apache Thu, 09 Aug 2007 13:23:23 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-71375 Bravo! I just took a look at your robots.txt file and I would recommend taking a look at the updated <a href="http://www.askapache.com/seo/updated-robotstxt-for-wordpress.html" rel="nofollow">wordpress robots.txt file</a> on askapache. Bravo! I just took a look at your robots.txt file and I would recommend taking a look at the updated wordpress robots.txt file on askapache.

]]> By: Tuto de l’été n°3: créez un fichier robots.txt pour votre blog Wordpresshttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-71311 Tuto de l’été n°3: créez un fichier robots.txt pour votre blog Wordpress Tue, 07 Aug 2007 13:37:04 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-71311 [...] Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days, aussi de ConnectedInterne.... [...] [...] Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days, aussi de ConnectedInterne…. [...]

]]> By: Evertonhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-70732 Everton Sun, 29 Jul 2007 09:31:07 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-70732 I think you need:User-agent: Googlebot Disallow: /*/welcome/$ User-agent: * Disallow: /news/welcome/Hoepfully someone else will confirm I think you need:

User-agent: Googlebot
Disallow: /*/welcome/$
User-agent: *
Disallow: /news/welcome/

Hoepfully someone else will confirm

]]> By: Maxhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-70685 Max Fri, 27 Jul 2007 19:11:42 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-70685 If I want to block a subdirectory only. e.g. http://www.site.com/news/welcomeWhat command should I sue:1. Disallow: /news/welcome/ 2. Disallow: /*/welcome/ 3. Disallow: /welcomeI know number 1 should be 100% works. But sometimes, I don't want to type the exact path.As for number 2, I think it works for google and yahoo bot only. I only these 2 which accept "*".As for number 3, I don't know whether it works or not. As for as I understand, with or without "/" make a big differences. With "/", you are saying to that directory only. Without "/", you are saying anything start from /welcome. Please correct me if I am wrong.So the subdirectory news/welcome also start from /welcome, so I can't sure whether Disallow: /welcome will block this subdirectory or not. If I want to block a subdirectory only. e.g. http://www.site.com/news/welcome

What command should I sue:

1. Disallow: /news/welcome/
2. Disallow: /*/welcome/
3. Disallow: /welcome

I know number 1 should be 100% works. But sometimes, I don’t want to type the exact path.

As for number 2, I think it works for google and yahoo bot only. I only these 2 which accept “*”.

As for number 3, I don’t know whether it works or not. As for as I understand, with or without “/” make a big differences. With “/”, you are saying to that directory only. Without “/”, you are saying anything start from /welcome. Please correct me if I am wrong.

So the subdirectory news/welcome also start from /welcome, so I can’t sure whether Disallow: /welcome will block this subdirectory or not.

]]> By: Ignore Traffic for More Traffics! | Blogging on Tech, Web 2.0 and Money Making Tipshttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-69829 Ignore Traffic for More Traffics! | Blogging on Tech, Web 2.0 and Money Making Tips Tue, 10 Jul 2007 00:21:05 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-69829 [...] Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days [...] [...] Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days [...]

]]> By: clickden.net » Blog Archive » BTTB DL Linkshttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-69502 clickden.net » Blog Archive » BTTB DL Links Tue, 03 Jul 2007 01:40:06 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-69502 [...] Finally, get a robots.txt file set up for the search engine crawlers. I use one I found at Connected internet Blog. Google and other crawlers expect to find a robots.txt in your root directory. Many of the error [...] [...] Finally, get a robots.txt file set up for the search engine crawlers. I use one I found at Connected internet Blog. Google and other crawlers expect to find a robots.txt in your root directory. Many of the error [...]

]]> By: Zathhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-4/#comment-69348 Zath Sat, 30 Jun 2007 18:49:54 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-69348 <p>I ended up putting the robots file back and now the supplemental links has been updated, mine dropped from around 2000 to 250! Whether that will make a big difference to my ranking and traffic, we'll see, but it sure sounds good in terms of supplemental links for what it's worth.</p> I ended up putting the robots file back and now the supplemental links has been updated, mine dropped from around 2000 to 250! Whether that will make a big difference to my ranking and traffic, we’ll see, but it sure sounds good in terms of supplemental links for what it’s worth.

]]> By: Increase Traffic With Robots.txt | WordPress SEOhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-4/#comment-69082 Increase Traffic With Robots.txt | WordPress SEO Tue, 26 Jun 2007 03:24:29 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-69082 [...] amazing how a simple file can drastically affect search engine optimization and traffic. Here is a great example of just that. Letting Google know what not to index can really impact the number [...] [...] amazing how a simple file can drastically affect search engine optimization and traffic. Here is a great example of just that. Letting Google know what not to index can really impact the number [...]

]]> By: Evertonhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-4/#comment-69058 Everton Mon, 25 Jun 2007 18:07:55 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-69058 @Ana1. Looking at the sahre this url it looks like the structure is: sitename/?p=1923&akst_action=share-this, so <code>Disallow: /?p=$</code> shoudl do the trick I think2.Not sure - anyone else?3. Yes @Ana

1. Looking at the sahre this url it looks like the structure is: sitename/?p=1923&akst_action=share-this, so Disallow: /?p=$ shoudl do the trick I think

2.Not sure – anyone else?

3. Yes

]]> By: Ana Luizahttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-4/#comment-69056 Ana Luiza Mon, 25 Jun 2007 17:23:19 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-69056 hi Everton, I love this article, I've been giving the robots issue some thought but have been mostly confused till now. I will probably give your robots file a try on my blog soon.I 've got a few questions - hopefully you'll have the time to see them:1. I have currently 732 supp pages (on my other blog) and most of them are Share-this pages generated by the plugin Share this. I'm thinking it'd be a good idea to unindex these. What's your opinion, and how do I include this piece of instruction in robots.txt?2. I've read that unindexing feeds will prevent my blog from being included in Google Blog Search. Do you know if that's true?3. What kind of Wordpress page has /date/ in the permalink? Is it daily archive pages?Thank you, and I really appreciate the way you share your discoveries Ana hi Everton,
I love this article, I’ve been giving the robots issue some thought but have been mostly confused till now. I will probably give your robots file a try on my blog soon.

I ‘ve got a few questions – hopefully you’ll have the time to see them:

1. I have currently 732 supp pages (on my other blog) and most of them are Share-this pages generated by the plugin Share this. I’m thinking it’d be a good idea to unindex these. What’s your opinion, and how do I include this piece of instruction in robots.txt?

2. I’ve read that unindexing feeds will prevent my blog from being included in Google Blog Search. Do you know if that’s true?

3. What kind of Wordpress page has /date/ in the permalink? Is it daily archive pages?

Thank you, and I really appreciate the way you share your discoveries
Ana

]]> By: Evertonhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-4/#comment-67882 Everton Thu, 14 Jun 2007 12:13:54 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-67882 @EternalsoulI've never used blogger, so I wouldn't know.@tomadding a robots.txt file should be useful for all sites. What you need to do is go through your site, and think through what pages you want appearing and which you don't. You can also use ‘site:www.YOURDOMAIN.COM -view ***‘ to see what pages are currently appearing the index, to see what things Google is currently picking up that you rather they didn't @Eternalsoul

I’ve never used blogger, so I wouldn’t know.

@tom

adding a robots.txt file should be useful for all sites. What you need to do is go through your site, and think through what pages you want appearing and which you don’t. You can also use ‘site:www.YOURDOMAIN.COM -view ***‘ to see what pages are currently appearing the index, to see what things Google is currently picking up that you rather they didn’t

]]> By: Eternalsoulhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-4/#comment-67881 Eternalsoul Thu, 14 Jun 2007 12:05:25 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-67881 Hi Everton I am a newbie to blogging. I blog from the new blogger platform. Is it possible to add the robots.txt in that and if so, can you please tell me how to do so? As of now, I think I have nil traffic. I would like to build it up and monetise it. <a href="http://soulfullfood.blogspot.com" rel="nofollow">Eternalsoul</a> Hi Everton
I am a newbie to blogging. I blog from the new blogger platform. Is it possible to add the robots.txt in that and if so, can you please tell me how to do so? As of now, I think I have nil traffic. I would like to build it up and monetise it.
Eternalsoul

]]> By: tomhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-4/#comment-67874 tom Thu, 14 Jun 2007 11:18:39 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-67874 Hi... Have just built my very first site and desperately trying to get it ranked etc.. haven't got a clue what robots text to use.. the theory I've read above all sounds great but all seems to relate to blogs?? Although I will at some point I guess, I don't yet use blogs..I'm sure you guys will spot numerous errors with my 1st site so be kind!! main problem at the moment is I have several pages in the main index but a hell of a lot in the supplementary results... including some of my main pages!I need a basic robots.txt to get the stuff out the supplementary into the main index...I currently have no robots.txt at allwww.jdelectricalgrantham.co.ukHelp would be very much appreciated.cheers Hi… Have just built my very first site and desperately trying to get it ranked etc.. haven’t got a clue what robots text to use.. the theory I’ve read above all sounds great but all seems to relate to blogs?? Although I will at some point I guess, I don’t yet use blogs..

I’m sure you guys will spot numerous errors with my 1st site so be kind!! main problem at the moment is I have several pages in the main index but a hell of a lot in the supplementary results… including some of my main pages!

I need a basic robots.txt to get the stuff out the supplementary into the main index…

I currently have no robots.txt at all

http://www.jdelectricalgrantham.co.uk

Help would be very much appreciated.

cheers

]]> By: Ravihttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-4/#comment-66314 Ravi Mon, 28 May 2007 16:13:19 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-66314 I have done the same for mine. Waiting for the result I have done the same for mine. Waiting for the result

]]> By: Glennhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-4/#comment-64526 Glenn Sun, 13 May 2007 17:35:47 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-64526 Anyone have any thoughtst about disallowing the following directories in addition to the ones recommended in this post?Disallow: /index.php Disallow: /category/Here is the post about why these are included?http://www.earnersblog.com/removing-supplementals-wordpress/ Anyone have any thoughtst about disallowing the following directories in addition to the ones recommended in this post?

Disallow: /index.php
Disallow: /category/

Here is the post about why these are included?

http://www.earnersblog.com/removing-supplementals-wordpress/

]]> By: Glennhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-4/#comment-64504 Glenn Sun, 13 May 2007 13:56:06 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-64504 I just discovered my feed is not at www.mysite.com/blog/feed but at www.mysite.com/blog/index.php/feed. So I changed my robots.txt file accordingly. Please Blog guru of the group let me know if it's correct. My site is www.SpyGearCo.com/blogUser-agent: Googlebot Disallow: /blog/index.php/feed/$ Disallow: /blog/index.php/feed/rss/$ Disallow: /blog/index.php/trackback/$ User-agent: * Disallow: /blog/index.php/wp- Disallow: /blog/index.php/feed/ Disallow: /blog/index.php/trackback/ Disallow: /blog/index.php/rss/ Disallow: /blog/index.php/comments/feed/ Disallow: /blog/index.php/page/ Disallow: /blog/index.php/date/ Disallow: /blog/index.php/comments/ Disallow: /rsscb I just discovered my feed is not at http://www.mysite.com/blog/feed but at http://www.mysite.com/blog/index.php/feed. So I changed my robots.txt file accordingly. Please Blog guru of the group let me know if it’s correct. My site is http://www.SpyGearCo.com/blog

User-agent: Googlebot
Disallow: /blog/index.php/feed/$
Disallow: /blog/index.php/feed/rss/$
Disallow: /blog/index.php/trackback/$
User-agent: *
Disallow: /blog/index.php/wp-
Disallow: /blog/index.php/feed/
Disallow: /blog/index.php/trackback/
Disallow: /blog/index.php/rss/
Disallow: /blog/index.php/comments/feed/
Disallow: /blog/index.php/page/
Disallow: /blog/index.php/date/
Disallow: /blog/index.php/comments/
Disallow: /rsscb

]]> By: Zathhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-3/#comment-64458 Zath Sun, 13 May 2007 09:03:52 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-64458 In that case Everton, I'll put it back, perhaps it was just a co-incidence.I'll see how it goes in the next couple of weeks. Cheers! In that case Everton, I’ll put it back, perhaps it was just a co-incidence.

I’ll see how it goes in the next couple of weeks. Cheers!

]]> By: Evertonhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-3/#comment-64437 Everton Sun, 13 May 2007 06:50:29 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-64437 @ZathNothing in the robots.txt file I posted should decrease your Google traffic as it is pretty basic.@GlennCan't see any problems. Maybe you need to wait a bit longer @Zath

Nothing in the robots.txt file I posted should decrease your Google traffic as it is pretty basic.

@Glenn

Can’t see any problems. Maybe you need to wait a bit longer

]]> By: Glennhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-3/#comment-64338 Glenn Sat, 12 May 2007 11:58:22 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-64338 No change after about 6 days. My blog is in a subdirectory called blog (i.e. www.MySite.com/blog) on my main site. Here is the contents of my robots.txt file.Anyone have any ideas? Thanks.User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ User-agent: * Disallow: /blog/wp- Disallow: /blog/feed/ Disallow: /blog/trackback/ Disallow: /blog/rss/ Disallow: /blog/comments/feed/ Disallow: /blog/page/ Disallow: /blog/date/ Disallow: /blog/comments/ Disallow: /rsscb No change after about 6 days. My blog is in a subdirectory called blog (i.e. http://www.MySite.com/blog) on my main site. Here is the contents of my robots.txt file.

Anyone have any ideas? Thanks.

User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
User-agent: *
Disallow: /blog/wp-
Disallow: /blog/feed/
Disallow: /blog/trackback/
Disallow: /blog/rss/
Disallow: /blog/comments/feed/
Disallow: /blog/page/
Disallow: /blog/date/
Disallow: /blog/comments/
Disallow: /rsscb

]]> By: Zathhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-3/#comment-64300 Zath Sat, 12 May 2007 08:40:41 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-64300 This could be completely unrelated, but about a week after I added the above basic robots.txt file, my search engine traffic seems to have dropped by around 50%Could this have anything to do with the robots.txt - or just a big co-incidence?For the time being I've removed the robots.txt to see if my traffic levels go back up. This could be completely unrelated, but about a week after I added the above basic robots.txt file, my search engine traffic seems to have dropped by around 50%

Could this have anything to do with the robots.txt – or just a big co-incidence?

For the time being I’ve removed the robots.txt to see if my traffic levels go back up.

]]> By: Evertonhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-3/#comment-63595 Everton Mon, 07 May 2007 17:23:39 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-63595 - open notepad - paste the text you want to use - save the file as robots.txt - upload it to your blog's root directory - open notepad
- paste the text you want to use
- save the file as robots.txt
- upload it to your blog’s root directory

]]> By: Vikramhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-3/#comment-63588 Vikram Mon, 07 May 2007 16:25:05 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-63588 can anyone help me on how to install the Robots.txt file ... Please help mePlease reply to vicky316[at]gmail[dot]com can anyone help me on how to install the Robots.txt file … Please help me

Please reply to vicky316[at]gmail[dot]com

]]> By: Glennhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-3/#comment-63436 Glenn Sun, 06 May 2007 21:24:22 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-63436 What does the below command do?Disallow: /wp-Does it disallow all directories that start with wp- in the WordPress blog?I too got bit by Google around 5/1/07 and nealy all of my blog posts went into the Supplemental result decreasing my traffic and sales overall.Hope it works. What does the below command do?

Disallow: /wp-

Does it disallow all directories that start with wp- in the WordPress blog?

I too got bit by Google around 5/1/07 and nealy all of my blog posts went into the Supplemental result decreasing my traffic and sales overall.

Hope it works.

]]> By: Zathhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-3/#comment-63341 Zath Sat, 05 May 2007 21:13:21 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-63341 I've now added a robots.txt file to my site, it's only basic one for wordpress blogs so far, but I'll see what, if any difference that makes to start with. I’ve now added a robots.txt file to my site, it’s only basic one for wordpress blogs so far, but I’ll see what, if any difference that makes to start with.

]]> By: Jenhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-3/#comment-62840 Jen Mon, 30 Apr 2007 23:54:03 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-62840 Hm, I haven't added a robots.txt file yet, but I think I will now! Thanks for the idea. Hm, I haven’t added a robots.txt file yet, but I think I will now! Thanks for the idea.

]]> By: moseyhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-3/#comment-62556 mosey Sat, 28 Apr 2007 15:34:38 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-62556 I also run a forum as part of my site and have found quite alot of the forum links in the supplementary search... does this mean I should add /forum to the robots.txt? A little bit confused. I also run a forum as part of my site and have found quite alot of the forum links in the supplementary search… does this mean I should add /forum to the robots.txt? A little bit confused.

]]> By: Eddiehttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-2/#comment-62512 Eddie Sat, 28 Apr 2007 08:02:14 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-62512 <a href="http://ghacks.net" rel="nofollow">Martin</a> told me as well about the robots.txt and this article.Although my site is still young and I have <em>only</em> 31 supplemental results (I was shocked that one of my main articles was among them). So I added the robots.txt a moment ago.Thank you for that great tip! Eddie Martin told me as well about the robots.txt and this article.

Although my site is still young and I have only 31 supplemental results (I was shocked that one of my main articles was among them). So I added the robots.txt a moment ago.

Thank you for that great tip!
Eddie

]]> By: Revisiting robots.txt | Twenty Stepshttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-2/#comment-61754 Revisiting robots.txt | Twenty Steps Wed, 18 Apr 2007 13:03:23 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-61754 [...] I’m going to give Plan B a whirl. Plan B is to replicate Evertons robots.txt file which has increased his Google traffic by 16% in 4 days. I’ll keep an eye on things through Google Webmaster Tools and let you know how things pan [...] [...] I’m going to give Plan B a whirl. Plan B is to replicate Evertons robots.txt file which has increased his Google traffic by 16% in 4 days. I’ll keep an eye on things through Google Webmaster Tools and let you know how things pan [...]

]]> By: Evertonhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-2/#comment-61386 Everton Sun, 15 Apr 2007 19:07:41 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-61386 @Ajaythe 16% increased overall traffic by around 10% @Ajay

the 16% increased overall traffic by around 10%

]]> By: Brown Baronhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-2/#comment-61278 Brown Baron Sat, 14 Apr 2007 12:13:20 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-61278 Finally added mine. Good thing I'm here regularly otherwise I'd forget a lot of things I was planning on doing. Finally added mine. Good thing I’m here regularly otherwise I’d forget a lot of things I was planning on doing.

]]> By: Ajayhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-2/#comment-61268 Ajay Sat, 14 Apr 2007 11:39:25 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-61268 Thanks for the suggestions Everton. I have a basic robots.txt that I use to disallow the bot from certain sections.How has it affected the overall traffic. i.e. the 16% increase has affected overall site traffic by how much? Thanks for the suggestions Everton. I have a basic robots.txt that I use to disallow the bot from certain sections.

How has it affected the overall traffic. i.e. the 16% increase has affected overall site traffic by how much?

]]> By: sourcerhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-2/#comment-61262 sourcer Sat, 14 Apr 2007 10:33:25 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-61262 I have already added a robots.txt file on my blog and I see more visits from google but I never calculated how much in % I have already added a robots.txt file on my blog and I see more visits from google but I never calculated how much in %

]]> By: Vijayhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-2/#comment-61259 Vijay Sat, 14 Apr 2007 10:18:27 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-61259 I've added a robots.txt. It did help in filtering out google indexing my login form and the like.. I guess it would be useful I’ve added a robots.txt. It did help in filtering out google indexing my login form and the like..
I guess it would be useful

]]> By: Evertonhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-2/#comment-61154 Everton Fri, 13 Apr 2007 08:25:16 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-61154 I hope you get a few responses Dan as that's a long list. My gut tells me it's too long, but I'm afraid I'm not fully up to speed in this area yet. I hope you get a few responses Dan as that’s a long list. My gut tells me it’s too long, but I’m afraid I’m not fully up to speed in this area yet.

]]> By: Dan and Jenniferhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-2/#comment-61147 Dan and Jennifer Fri, 13 Apr 2007 02:29:21 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-61147 Ok, I've just spent a whole bunch of time with my new friend from Google Webmaster tools - the robots.txt analysis tool. You paste the content of your new robots.txt and provide him a test URL and he tells you his results based on GoogleBot, AdSenseBot, and the others.Anyway, here's the new file I came up with, WAY longer thank I'd initially hoped for.But does cover some important items, like the "tracback" and comment feed for each post as far as GoogleBot is concerned.Does anyone see ANYTHING wrong with this file?Thanks! Dan----- # Robots.txt file# All robots will spider the domain User-agent: * Disallow: /Openads/ Disallow: /wp- Disallow: /feed/ Disallow: /rss/ Disallow: /trackback/ Disallow: /comments/feed/ Disallow: /wp-content/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /cgi-bin/ Disallow: /styles/ Disallow: /dnld/ Allow: /wp-content/uploads/ Allow: /wp-content/themes/mistylook/img# GoogleBot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ Disallow: /Openads/ Disallow: /wp- Disallow: /feed/ Disallow: /rss/ Disallow: /trackback/ Disallow: /comments/feed/ Disallow: /wp-content/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /cgi-bin/ Disallow: /styles/ Disallow: /dnld/ Allow: /wp-content/uploads/ Allow: /wp-content/themes/mistylook/img# allow adsense bot on entire site User-agent: Mediapartners-Google Allow: /* # allow google image bot to search all images User-agent: Googlebot-Image Allow: /*# allow AdWords PPC bot on entire site User-agent: Adsbot-Google Allow: /* Ok, I’ve just spent a whole bunch of time with my new friend from Google Webmaster tools – the robots.txt analysis tool. You paste the content of your new robots.txt and provide him a test URL and he tells you his results based on GoogleBot, AdSenseBot, and the others.

Anyway, here’s the new file I came up with, WAY longer thank I’d initially hoped for.

But does cover some important items, like the “tracback” and comment feed for each post as far as GoogleBot is concerned.

Does anyone see ANYTHING wrong with this file?

Thanks!
Dan

—–
# Robots.txt file

# All robots will spider the domain
User-agent: *
Disallow: /Openads/
Disallow: /wp-
Disallow: /feed/
Disallow: /rss/
Disallow: /trackback/
Disallow: /comments/feed/
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /cgi-bin/
Disallow: /styles/
Disallow: /dnld/
Allow: /wp-content/uploads/
Allow: /wp-content/themes/mistylook/img

# GoogleBot
User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /Openads/
Disallow: /wp-
Disallow: /feed/
Disallow: /rss/
Disallow: /trackback/
Disallow: /comments/feed/
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /cgi-bin/
Disallow: /styles/
Disallow: /dnld/
Allow: /wp-content/uploads/
Allow: /wp-content/themes/mistylook/img

# allow adsense bot on entire site
User-agent: Mediapartners-Google
Allow: /*

# allow google image bot to search all images
User-agent: Googlebot-Image
Allow: /*

# allow AdWords PPC bot on entire site
User-agent: Adsbot-Google
Allow: /*

]]> By: Dan and Jenniferhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-2/#comment-61146 Dan and Jennifer Fri, 13 Apr 2007 02:19:16 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-61146 Hey Everton,Thanks for bringing this up - we've been runing with an old bare-bones robots.txt file which paid no attention to wordpress feeds or other such features.I wonder if it's the robots.txt file change that increased your traffic, or coincidentally the upcoming pagerank change.We've noticed a very significant increase in Google traffic the past two weeks, which is very similar in magnitude to the traffic increase right before the last pagerank upgrade.Have an awesome day! Dan Hey Everton,

Thanks for bringing this up – we’ve been runing with an old bare-bones robots.txt file which paid no attention to wordpress feeds or other such features.

I wonder if it’s the robots.txt file change that increased your traffic, or coincidentally the upcoming pagerank change.

We’ve noticed a very significant increase in Google traffic the past two weeks, which is very similar in magnitude to the traffic increase right before the last pagerank upgrade.

Have an awesome day!
Dan

]]> By: moseyhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-1/#comment-61124 mosey Thu, 12 Apr 2007 19:33:03 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-61124 Thanks for this tip :) I never realised the significance of robots.txt! I currently have 158 supplementary links so will see if this reduces it. I do already get a fair amount of traffic from google (as the terms are specific to mysite) but maybe this will make it even better? I'm confused about what robots.txt does ... does this mean that the supplementary pages become main pages? Thanks for this tip :) I never realised the significance of robots.txt! I currently have 158 supplementary links so will see if this reduces it. I do already get a fair amount of traffic from google (as the terms are specific to mysite) but maybe this will make it even better?
I’m confused about what robots.txt does … does this mean that the supplementary pages become main pages?

]]> By: Ashwinhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-1/#comment-61081 Ashwin Thu, 12 Apr 2007 12:00:03 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-61081 Thanks for the help. I've uploaded. Let's wait for the result. Thanks for the help. I’ve uploaded. Let’s wait for the result.

]]> By: Zanehttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-1/#comment-61077 Zane Thu, 12 Apr 2007 11:15:52 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-61077 well, i'll give it go. sounds good, thanks for sharing with us well, i’ll give it go. sounds good, thanks for sharing with us

]]> By: Evertonhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-1/#comment-61051 Everton Thu, 12 Apr 2007 08:51:50 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-61051 @AshwinYes. The directory with your wp-admin etc folders is your root directory.No question is a newbie question - always feel free to fire off anything on this site. We're all friends here! @Ashwin

Yes. The directory with your wp-admin etc folders is your root directory.

No question is a newbie question – always feel free to fire off anything on this site. We’re all friends here!

]]> By: Ashwinhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-1/#comment-61044 Ashwin Thu, 12 Apr 2007 08:30:06 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-61044 Sorry to bother you with noobish questions. But can you tell me where I have to upload. I have my WP in the "/" place only. So shall I upload it with the place where I have some folders like wp-admin, wp-content and things like that or should I upload to a folder before that. Sorry to bother you with noobish questions. But can you tell me where I have to upload. I have my WP in the “/” place only. So shall I upload it with the place where I have some folders like wp-admin, wp-content and things like that or should I upload to a folder before that.

]]> By: Martinhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-1/#comment-61010 Martin Thu, 12 Apr 2007 07:09:49 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-61010 It took a while longer for my supplemental results to decrease and I heard that it could take a long time for some. Mine dropped from 12600 to 50 in the last days which is really nice. It took a while longer for my supplemental results to decrease and I heard that it could take a long time for some. Mine dropped from 12600 to 50 in the last days which is really nice.

]]> By: Mr.Bytehttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-1/#comment-61007 Mr.Byte Thu, 12 Apr 2007 06:04:36 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-61007 Yeah I also have added but I am not sure whether to remove tags page or not. I think at least I should change my posts in tags page to excerpt rather than what it looks like in index page. Yeah I also have added but I am not sure whether to remove tags page or not. I think at least I should change my posts in tags page to excerpt rather than what it looks like in index page.

]]> By: Evertonhttp://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-1/#comment-61005 Everton Thu, 12 Apr 2007 06:00:28 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-61005 @AlpeshCorrect@AshwinCreating a file is simple. Just create a new text file in notepad containing your entries and then upload it to the root directory of your site @Alpesh

Correct

@Ashwin

Creating a file is simple. Just create a new text file in notepad containing your entries and then upload it to the root directory of your site

]]>