Comments on: Adding A Robots.txt File Has Increased My Google Traffic By 16% In 4 Days http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/ Updated daily with the latest news, tips, tweaks, social networking, wordpress tips & tweaks, windows optimization help and more. Fri, 10 Feb 2012 21:47:03 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: NPO http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-9/#comment-299508 NPO Sun, 17 Jul 2011 19:34:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-299508 I have some 2100 supplemental pages..added the robots.txt now..hopefully it should do some good for my blog's seo I have some 2100 supplemental pages..added the robots.txt now..hopefully it should do some good for my blog’s seo

]]> By: Helmut blockheizkraftwerk http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-9/#comment-188712 Helmut blockheizkraftwerk Thu, 19 Aug 2010 20:01:33 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-188712 great! great!

]]> By: patrick http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-9/#comment-155490 patrick Thu, 12 Feb 2009 03:37:58 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-155490 Do you think this works on a none word press site? Do you think this works on a none word press site?

]]> By: patrick http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-9/#comment-222079 patrick Thu, 12 Feb 2009 03:37:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222079 Do you think this works on a none word press site? Do you think this works on a none word press site?

]]> By: Webrecsol Optimizer http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-9/#comment-155278 Webrecsol Optimizer Wed, 21 Jan 2009 05:47:11 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-155278 hello friend, i rode your article as a robots.txt and you say that by robots file increase the traffic then can u clearly told me how to possible... hello friend, i rode your article as a robots.txt and you say that by robots file increase the traffic then can u clearly told me how to possible…

]]> By: Webrecsol Optimizer http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-9/#comment-222078 Webrecsol Optimizer Wed, 21 Jan 2009 05:47:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222078 hello friend, i rode your article as a robots.txt and you say that by robots file increase the traffic then can u clearly told me how to possible... hello friend, i rode your article as a robots.txt and you say that by robots file increase the traffic then can u clearly told me how to possible…

]]> By: FreelancersAtLarge.com http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-9/#comment-88019 FreelancersAtLarge.com Sun, 04 Nov 2007 08:32:04 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-88019 About three weeks ago, I finally got around to uploading the robots.txt file for www.freelancersatlarge.com (it had been way down on the list of prorities). In three weeks, the number of links in Google's index increased by 5 times! About three weeks ago, I finally got around to uploading the robots.txt file for http://www.freelancersatlarge.com (it had been way down on the list of prorities).

In three weeks, the number of links in Google’s index increased by 5 times!

]]> By: FreelancersAtLarge.com http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-9/#comment-222077 FreelancersAtLarge.com Sun, 04 Nov 2007 08:32:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222077 About three weeks ago, I finally got around to uploading the robots.txt file for www.freelancersatlarge.com (it had been way down on the list of prorities). In three weeks, the number of links in Google's index increased by 5 times! About three weeks ago, I finally got around to uploading the robots.txt file for http://www.freelancersatlarge.com (it had been way down on the list of prorities).

In three weeks, the number of links in Google’s index increased by 5 times!

]]> By: Ask Apache http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-9/#comment-71375 Ask Apache Thu, 09 Aug 2007 13:23:23 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-71375 Bravo! I just took a look at your robots.txt file and I would recommend taking a look at the updated <a href="http://www.askapache.com/seo/updated-robotstxt-for-wordpress.html" rel="nofollow">wordpress robots.txt file</a> on askapache. Bravo! I just took a look at your robots.txt file and I would recommend taking a look at the updated wordpress robots.txt file on askapache.

]]> By: Ask Apache http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-9/#comment-222076 Ask Apache Thu, 09 Aug 2007 13:23:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222076 Bravo! I just took a look at your robots.txt file and I would recommend taking a look at the updated <a href="http://www.askapache.com/seo/updated-robotstxt-for-wordpress.html" rel="nofollow">wordpress robots.txt file</a> on askapache. Bravo! I just took a look at your robots.txt file and I would recommend taking a look at the updated wordpress robots.txt file on askapache.

]]> By: Everton http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-8/#comment-70732 Everton Sun, 29 Jul 2007 09:31:07 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-70732 I think you need: User-agent: Googlebot Disallow: /*/welcome/$ User-agent: * Disallow: /news/welcome/ Hoepfully someone else will confirm I think you need:

User-agent: Googlebot
Disallow: /*/welcome/$
User-agent: *
Disallow: /news/welcome/

Hoepfully someone else will confirm

]]> By: Everton http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-8/#comment-222075 Everton Sun, 29 Jul 2007 09:31:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222075 I think you need: User-agent: Googlebot Disallow: /*/welcome/$ User-agent: * Disallow: /news/welcome/ Hoepfully someone else will confirm I think you need:

User-agent: Googlebot
Disallow: /*/welcome/$
User-agent: *
Disallow: /news/welcome/

Hoepfully someone else will confirm

]]> By: Max http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-8/#comment-70685 Max Fri, 27 Jul 2007 19:11:42 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-70685 If I want to block a subdirectory only. e.g. http://www.site.com/news/welcome What command should I sue: 1. Disallow: /news/welcome/ 2. Disallow: /*/welcome/ 3. Disallow: /welcome I know number 1 should be 100% works. But sometimes, I don't want to type the exact path. As for number 2, I think it works for google and yahoo bot only. I only these 2 which accept "*". As for number 3, I don't know whether it works or not. As for as I understand, with or without "/" make a big differences. With "/", you are saying to that directory only. Without "/", you are saying anything start from /welcome. Please correct me if I am wrong. So the subdirectory news/welcome also start from /welcome, so I can't sure whether Disallow: /welcome will block this subdirectory or not. If I want to block a subdirectory only. e.g. http://www.site.com/news/welcome

What command should I sue:

1. Disallow: /news/welcome/
2. Disallow: /*/welcome/
3. Disallow: /welcome

I know number 1 should be 100% works. But sometimes, I don’t want to type the exact path.

As for number 2, I think it works for google and yahoo bot only. I only these 2 which accept “*”.

As for number 3, I don’t know whether it works or not. As for as I understand, with or without “/” make a big differences. With “/”, you are saying to that directory only. Without “/”, you are saying anything start from /welcome. Please correct me if I am wrong.

So the subdirectory news/welcome also start from /welcome, so I can’t sure whether Disallow: /welcome will block this subdirectory or not.

]]> By: Max http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-8/#comment-222074 Max Fri, 27 Jul 2007 19:11:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222074 If I want to block a subdirectory only. e.g. http://www.site.com/news/welcome What command should I sue: 1. Disallow: /news/welcome/ 2. Disallow: /*/welcome/ 3. Disallow: /welcome I know number 1 should be 100% works. But sometimes, I don't want to type the exact path. As for number 2, I think it works for google and yahoo bot only. I only these 2 which accept "*". As for number 3, I don't know whether it works or not. As for as I understand, with or without "/" make a big differences. With "/", you are saying to that directory only. Without "/", you are saying anything start from /welcome. Please correct me if I am wrong. So the subdirectory news/welcome also start from /welcome, so I can't sure whether Disallow: /welcome will block this subdirectory or not. If I want to block a subdirectory only. e.g. http://www.site.com/news/welcome

What command should I sue:

1. Disallow: /news/welcome/
2. Disallow: /*/welcome/
3. Disallow: /welcome

I know number 1 should be 100% works. But sometimes, I don’t want to type the exact path.

As for number 2, I think it works for google and yahoo bot only. I only these 2 which accept “*”.

As for number 3, I don’t know whether it works or not. As for as I understand, with or without “/” make a big differences. With “/”, you are saying to that directory only. Without “/”, you are saying anything start from /welcome. Please correct me if I am wrong.

So the subdirectory news/welcome also start from /welcome, so I can’t sure whether Disallow: /welcome will block this subdirectory or not.

]]> By: Zath http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-8/#comment-69348 Zath Sat, 30 Jun 2007 18:49:54 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-69348 <p>I ended up putting the robots file back and now the supplemental links has been updated, mine dropped from around 2000 to 250! Whether that will make a big difference to my ranking and traffic, we'll see, but it sure sounds good in terms of supplemental links for what it's worth.</p> I ended up putting the robots file back and now the supplemental links has been updated, mine dropped from around 2000 to 250! Whether that will make a big difference to my ranking and traffic, we’ll see, but it sure sounds good in terms of supplemental links for what it’s worth.

]]> By: Zath http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-8/#comment-222073 Zath Sat, 30 Jun 2007 18:49:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222073 I ended up putting the robots file back and now the supplemental links has been updated, mine dropped from around 2000 to 250! Whether that will make a big difference to my ranking and traffic, we'll see, but it sure sounds good in terms of supplemental links for what it's worth. I ended up putting the robots file back and now the supplemental links has been updated, mine dropped from around 2000 to 250! Whether that will make a big difference to my ranking and traffic, we’ll see, but it sure sounds good in terms of supplemental links for what it’s worth.

]]> By: Everton http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-8/#comment-69058 Everton Mon, 25 Jun 2007 18:07:55 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-69058 @Ana 1. Looking at the sahre this url it looks like the structure is: sitename/?p=1923&akst_action=share-this, so <code>Disallow: /?p=$</code> shoudl do the trick I think 2.Not sure - anyone else? 3. Yes @Ana

1. Looking at the sahre this url it looks like the structure is: sitename/?p=1923&akst_action=share-this, so Disallow: /?p=$ shoudl do the trick I think

2.Not sure – anyone else?

3. Yes

]]> By: Everton http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-8/#comment-222072 Everton Mon, 25 Jun 2007 18:07:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222072 @Ana 1. Looking at the sahre this url it looks like the structure is: sitename/?p=1923&akst;_action=share-this, so <code>Disallow: /?p=$</code> shoudl do the trick I think 2.Not sure - anyone else? 3. Yes @Ana

1. Looking at the sahre this url it looks like the structure is: sitename/?p=1923&akst;_action=share-this, so Disallow: /?p=$ shoudl do the trick I think

2.Not sure – anyone else?

3. Yes

]]> By: Ana Luiza http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-8/#comment-69056 Ana Luiza Mon, 25 Jun 2007 17:23:19 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-69056 hi Everton, I love this article, I've been giving the robots issue some thought but have been mostly confused till now. I will probably give your robots file a try on my blog soon. I 've got a few questions - hopefully you'll have the time to see them: 1. I have currently 732 supp pages (on my other blog) and most of them are Share-this pages generated by the plugin Share this. I'm thinking it'd be a good idea to unindex these. What's your opinion, and how do I include this piece of instruction in robots.txt? 2. I've read that unindexing feeds will prevent my blog from being included in Google Blog Search. Do you know if that's true? 3. What kind of Wordpress page has /date/ in the permalink? Is it daily archive pages? Thank you, and I really appreciate the way you share your discoveries Ana hi Everton,
I love this article, I’ve been giving the robots issue some thought but have been mostly confused till now. I will probably give your robots file a try on my blog soon.

I ‘ve got a few questions – hopefully you’ll have the time to see them:

1. I have currently 732 supp pages (on my other blog) and most of them are Share-this pages generated by the plugin Share this. I’m thinking it’d be a good idea to unindex these. What’s your opinion, and how do I include this piece of instruction in robots.txt?

2. I’ve read that unindexing feeds will prevent my blog from being included in Google Blog Search. Do you know if that’s true?

3. What kind of WordPress page has /date/ in the permalink? Is it daily archive pages?

Thank you, and I really appreciate the way you share your discoveries
Ana

]]> By: Ana Luiza http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-8/#comment-222071 Ana Luiza Mon, 25 Jun 2007 17:23:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222071 hi Everton, I love this article, I've been giving the robots issue some thought but have been mostly confused till now. I will probably give your robots file a try on my blog soon. I 've got a few questions - hopefully you'll have the time to see them: 1. I have currently 732 supp pages (on my other blog) and most of them are Share-this pages generated by the plugin Share this. I'm thinking it'd be a good idea to unindex these. What's your opinion, and how do I include this piece of instruction in robots.txt? 2. I've read that unindexing feeds will prevent my blog from being included in Google Blog Search. Do you know if that's true? 3. What kind of Wordpress page has /date/ in the permalink? Is it daily archive pages? Thank you, and I really appreciate the way you share your discoveries Ana hi Everton,
I love this article, I’ve been giving the robots issue some thought but have been mostly confused till now. I will probably give your robots file a try on my blog soon.

I ‘ve got a few questions – hopefully you’ll have the time to see them:

1. I have currently 732 supp pages (on my other blog) and most of them are Share-this pages generated by the plugin Share this. I’m thinking it’d be a good idea to unindex these. What’s your opinion, and how do I include this piece of instruction in robots.txt?

2. I’ve read that unindexing feeds will prevent my blog from being included in Google Blog Search. Do you know if that’s true?

3. What kind of WordPress page has /date/ in the permalink? Is it daily archive pages?

Thank you, and I really appreciate the way you share your discoveries
Ana

]]> By: Everton http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-7/#comment-67882 Everton Thu, 14 Jun 2007 12:13:54 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-67882 @Eternalsoul I've never used blogger, so I wouldn't know. @tom adding a robots.txt file should be useful for all sites. What you need to do is go through your site, and think through what pages you want appearing and which you don't. You can also use ‘site:www.YOURDOMAIN.COM -view ***‘ to see what pages are currently appearing the index, to see what things Google is currently picking up that you rather they didn't @Eternalsoul

I’ve never used blogger, so I wouldn’t know.

@tom

adding a robots.txt file should be useful for all sites. What you need to do is go through your site, and think through what pages you want appearing and which you don’t. You can also use ‘site:www.YOURDOMAIN.COM -view ***‘ to see what pages are currently appearing the index, to see what things Google is currently picking up that you rather they didn’t

]]> By: Everton http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-7/#comment-222070 Everton Thu, 14 Jun 2007 12:13:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222070 @Eternalsoul I've never used blogger, so I wouldn't know. @tom adding a robots.txt file should be useful for all sites. What you need to do is go through your site, and think through what pages you want appearing and which you don't. You can also use ‘site:www.YOURDOMAIN.COM -view ***‘ to see what pages are currently appearing the index, to see what things Google is currently picking up that you rather they didn't @Eternalsoul

I’ve never used blogger, so I wouldn’t know.

@tom

adding a robots.txt file should be useful for all sites. What you need to do is go through your site, and think through what pages you want appearing and which you don’t. You can also use ‘site:www.YOURDOMAIN.COM -view ***‘ to see what pages are currently appearing the index, to see what things Google is currently picking up that you rather they didn’t

]]> By: Eternalsoul http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-7/#comment-67881 Eternalsoul Thu, 14 Jun 2007 12:05:25 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-67881 Hi Everton I am a newbie to blogging. I blog from the new blogger platform. Is it possible to add the robots.txt in that and if so, can you please tell me how to do so? As of now, I think I have nil traffic. I would like to build it up and monetise it. <a href="http://soulfullfood.blogspot.com" rel="nofollow">Eternalsoul</a> Hi Everton
I am a newbie to blogging. I blog from the new blogger platform. Is it possible to add the robots.txt in that and if so, can you please tell me how to do so? As of now, I think I have nil traffic. I would like to build it up and monetise it.
Eternalsoul

]]> By: Eternalsoul http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-7/#comment-222069 Eternalsoul Thu, 14 Jun 2007 12:05:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222069 Hi Everton I am a newbie to blogging. I blog from the new blogger platform. Is it possible to add the robots.txt in that and if so, can you please tell me how to do so? As of now, I think I have nil traffic. I would like to build it up and monetise it. <a href="http://soulfullfood.blogspot.com" rel="nofollow">Eternalsoul</a> Hi Everton
I am a newbie to blogging. I blog from the new blogger platform. Is it possible to add the robots.txt in that and if so, can you please tell me how to do so? As of now, I think I have nil traffic. I would like to build it up and monetise it.
Eternalsoul

]]> By: tom http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-7/#comment-67874 tom Thu, 14 Jun 2007 11:18:39 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-67874 Hi... Have just built my very first site and desperately trying to get it ranked etc.. haven't got a clue what robots text to use.. the theory I've read above all sounds great but all seems to relate to blogs?? Although I will at some point I guess, I don't yet use blogs.. I'm sure you guys will spot numerous errors with my 1st site so be kind!! main problem at the moment is I have several pages in the main index but a hell of a lot in the supplementary results... including some of my main pages! I need a basic robots.txt to get the stuff out the supplementary into the main index... I currently have no robots.txt at all www.jdelectricalgrantham.co.uk Help would be very much appreciated. cheers Hi… Have just built my very first site and desperately trying to get it ranked etc.. haven’t got a clue what robots text to use.. the theory I’ve read above all sounds great but all seems to relate to blogs?? Although I will at some point I guess, I don’t yet use blogs..

I’m sure you guys will spot numerous errors with my 1st site so be kind!! main problem at the moment is I have several pages in the main index but a hell of a lot in the supplementary results… including some of my main pages!

I need a basic robots.txt to get the stuff out the supplementary into the main index…

I currently have no robots.txt at all

http://www.jdelectricalgrantham.co.uk

Help would be very much appreciated.

cheers

]]> By: tom http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-7/#comment-222068 tom Thu, 14 Jun 2007 11:18:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222068 Hi... Have just built my very first site and desperately trying to get it ranked etc.. haven't got a clue what robots text to use.. the theory I've read above all sounds great but all seems to relate to blogs?? Although I will at some point I guess, I don't yet use blogs.. I'm sure you guys will spot numerous errors with my 1st site so be kind!! main problem at the moment is I have several pages in the main index but a hell of a lot in the supplementary results... including some of my main pages! I need a basic robots.txt to get the stuff out the supplementary into the main index... I currently have no robots.txt at all www.jdelectricalgrantham.co.uk Help would be very much appreciated. cheers Hi… Have just built my very first site and desperately trying to get it ranked etc.. haven’t got a clue what robots text to use.. the theory I’ve read above all sounds great but all seems to relate to blogs?? Although I will at some point I guess, I don’t yet use blogs..

I’m sure you guys will spot numerous errors with my 1st site so be kind!! main problem at the moment is I have several pages in the main index but a hell of a lot in the supplementary results… including some of my main pages!

I need a basic robots.txt to get the stuff out the supplementary into the main index…

I currently have no robots.txt at all

http://www.jdelectricalgrantham.co.uk

Help would be very much appreciated.

cheers

]]> By: Ravi http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-7/#comment-66314 Ravi Mon, 28 May 2007 16:13:19 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-66314 I have done the same for mine. Waiting for the result I have done the same for mine. Waiting for the result

]]> By: Ravi http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-7/#comment-222067 Ravi Mon, 28 May 2007 16:13:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222067 I have done the same for mine. Waiting for the result I have done the same for mine. Waiting for the result

]]> By: Glenn http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-7/#comment-64526 Glenn Sun, 13 May 2007 17:35:47 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-64526 Anyone have any thoughtst about disallowing the following directories in addition to the ones recommended in this post? Disallow: /index.php Disallow: /category/ Here is the post about why these are included? http://www.earnersblog.com/removing-supplementals-wordpress/ Anyone have any thoughtst about disallowing the following directories in addition to the ones recommended in this post?

Disallow: /index.php
Disallow: /category/

Here is the post about why these are included?

http://www.earnersblog.com/removing-supplementals-wordpress/

]]> By: Glenn http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-7/#comment-222066 Glenn Sun, 13 May 2007 17:35:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222066 Anyone have any thoughtst about disallowing the following directories in addition to the ones recommended in this post? Disallow: /index.php Disallow: /category/ Here is the post about why these are included? http://www.earnersblog.com/removing-supplementals-wordpress/ Anyone have any thoughtst about disallowing the following directories in addition to the ones recommended in this post?

Disallow: /index.php
Disallow: /category/

Here is the post about why these are included?

http://www.earnersblog.com/removing-supplementals-wordpress/

]]> By: Glenn http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-6/#comment-64504 Glenn Sun, 13 May 2007 13:56:06 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-64504 I just discovered my feed is not at www.mysite.com/blog/feed but at www.mysite.com/blog/index.php/feed. So I changed my robots.txt file accordingly. Please Blog guru of the group let me know if it's correct. My site is www.SpyGearCo.com/blog User-agent: Googlebot Disallow: /blog/index.php/feed/$ Disallow: /blog/index.php/feed/rss/$ Disallow: /blog/index.php/trackback/$ User-agent: * Disallow: /blog/index.php/wp- Disallow: /blog/index.php/feed/ Disallow: /blog/index.php/trackback/ Disallow: /blog/index.php/rss/ Disallow: /blog/index.php/comments/feed/ Disallow: /blog/index.php/page/ Disallow: /blog/index.php/date/ Disallow: /blog/index.php/comments/ Disallow: /rsscb I just discovered my feed is not at http://www.mysite.com/blog/feed but at http://www.mysite.com/blog/index.php/feed. So I changed my robots.txt file accordingly. Please Blog guru of the group let me know if it’s correct. My site is http://www.SpyGearCo.com/blog

User-agent: Googlebot
Disallow: /blog/index.php/feed/$
Disallow: /blog/index.php/feed/rss/$
Disallow: /blog/index.php/trackback/$
User-agent: *
Disallow: /blog/index.php/wp-
Disallow: /blog/index.php/feed/
Disallow: /blog/index.php/trackback/
Disallow: /blog/index.php/rss/
Disallow: /blog/index.php/comments/feed/
Disallow: /blog/index.php/page/
Disallow: /blog/index.php/date/
Disallow: /blog/index.php/comments/
Disallow: /rsscb

]]> By: Glenn http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-6/#comment-222065 Glenn Sun, 13 May 2007 13:56:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222065 I just discovered my feed is not at www.mysite.com/blog/feed but at www.mysite.com/blog/index.php/feed. So I changed my robots.txt file accordingly. Please Blog guru of the group let me know if it's correct. My site is www.SpyGearCo.com/blog User-agent: Googlebot Disallow: /blog/index.php/feed/$ Disallow: /blog/index.php/feed/rss/$ Disallow: /blog/index.php/trackback/$ User-agent: * Disallow: /blog/index.php/wp- Disallow: /blog/index.php/feed/ Disallow: /blog/index.php/trackback/ Disallow: /blog/index.php/rss/ Disallow: /blog/index.php/comments/feed/ Disallow: /blog/index.php/page/ Disallow: /blog/index.php/date/ Disallow: /blog/index.php/comments/ Disallow: /rsscb I just discovered my feed is not at http://www.mysite.com/blog/feed but at http://www.mysite.com/blog/index.php/feed. So I changed my robots.txt file accordingly. Please Blog guru of the group let me know if it’s correct. My site is http://www.SpyGearCo.com/blog

User-agent: Googlebot
Disallow: /blog/index.php/feed/$
Disallow: /blog/index.php/feed/rss/$
Disallow: /blog/index.php/trackback/$
User-agent: *
Disallow: /blog/index.php/wp-
Disallow: /blog/index.php/feed/
Disallow: /blog/index.php/trackback/
Disallow: /blog/index.php/rss/
Disallow: /blog/index.php/comments/feed/
Disallow: /blog/index.php/page/
Disallow: /blog/index.php/date/
Disallow: /blog/index.php/comments/
Disallow: /rsscb

]]> By: Zath http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-6/#comment-64458 Zath Sun, 13 May 2007 09:03:52 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-64458 In that case Everton, I'll put it back, perhaps it was just a co-incidence. I'll see how it goes in the next couple of weeks. Cheers! In that case Everton, I’ll put it back, perhaps it was just a co-incidence.

I’ll see how it goes in the next couple of weeks. Cheers!

]]> By: Zath http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-6/#comment-222064 Zath Sun, 13 May 2007 09:03:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222064 In that case Everton, I'll put it back, perhaps it was just a co-incidence. I'll see how it goes in the next couple of weeks. Cheers! In that case Everton, I’ll put it back, perhaps it was just a co-incidence.

I’ll see how it goes in the next couple of weeks. Cheers!

]]> By: Everton http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-6/#comment-64437 Everton Sun, 13 May 2007 06:50:29 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-64437 @Zath Nothing in the robots.txt file I posted should decrease your Google traffic as it is pretty basic. @Glenn Can't see any problems. Maybe you need to wait a bit longer @Zath

Nothing in the robots.txt file I posted should decrease your Google traffic as it is pretty basic.

@Glenn

Can’t see any problems. Maybe you need to wait a bit longer

]]> By: Everton http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-6/#comment-222063 Everton Sun, 13 May 2007 06:50:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222063 @Zath Nothing in the robots.txt file I posted should decrease your Google traffic as it is pretty basic. @Glenn Can't see any problems. Maybe you need to wait a bit longer @Zath

Nothing in the robots.txt file I posted should decrease your Google traffic as it is pretty basic.

@Glenn

Can’t see any problems. Maybe you need to wait a bit longer

]]> By: Glenn http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-6/#comment-64338 Glenn Sat, 12 May 2007 11:58:22 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-64338 No change after about 6 days. My blog is in a subdirectory called blog (i.e. www.MySite.com/blog) on my main site. Here is the contents of my robots.txt file. Anyone have any ideas? Thanks. User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ User-agent: * Disallow: /blog/wp- Disallow: /blog/feed/ Disallow: /blog/trackback/ Disallow: /blog/rss/ Disallow: /blog/comments/feed/ Disallow: /blog/page/ Disallow: /blog/date/ Disallow: /blog/comments/ Disallow: /rsscb No change after about 6 days. My blog is in a subdirectory called blog (i.e. http://www.MySite.com/blog) on my main site. Here is the contents of my robots.txt file.

Anyone have any ideas? Thanks.

User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
User-agent: *
Disallow: /blog/wp-
Disallow: /blog/feed/
Disallow: /blog/trackback/
Disallow: /blog/rss/
Disallow: /blog/comments/feed/
Disallow: /blog/page/
Disallow: /blog/date/
Disallow: /blog/comments/
Disallow: /rsscb

]]> By: Glenn http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-6/#comment-222062 Glenn Sat, 12 May 2007 11:58:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222062 No change after about 6 days. My blog is in a subdirectory called blog (i.e. www.MySite.com/blog) on my main site. Here is the contents of my robots.txt file. Anyone have any ideas? Thanks. User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ User-agent: * Disallow: /blog/wp- Disallow: /blog/feed/ Disallow: /blog/trackback/ Disallow: /blog/rss/ Disallow: /blog/comments/feed/ Disallow: /blog/page/ Disallow: /blog/date/ Disallow: /blog/comments/ Disallow: /rsscb No change after about 6 days. My blog is in a subdirectory called blog (i.e. http://www.MySite.com/blog) on my main site. Here is the contents of my robots.txt file.

Anyone have any ideas? Thanks.

User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
User-agent: *
Disallow: /blog/wp-
Disallow: /blog/feed/
Disallow: /blog/trackback/
Disallow: /blog/rss/
Disallow: /blog/comments/feed/
Disallow: /blog/page/
Disallow: /blog/date/
Disallow: /blog/comments/
Disallow: /rsscb

]]> By: Zath http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-6/#comment-64300 Zath Sat, 12 May 2007 08:40:41 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-64300 This could be completely unrelated, but about a week after I added the above basic robots.txt file, my search engine traffic seems to have dropped by around 50% Could this have anything to do with the robots.txt - or just a big co-incidence? For the time being I've removed the robots.txt to see if my traffic levels go back up. This could be completely unrelated, but about a week after I added the above basic robots.txt file, my search engine traffic seems to have dropped by around 50%

Could this have anything to do with the robots.txt – or just a big co-incidence?

For the time being I’ve removed the robots.txt to see if my traffic levels go back up.

]]> By: Zath http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-6/#comment-222061 Zath Sat, 12 May 2007 08:40:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222061 This could be completely unrelated, but about a week after I added the above basic robots.txt file, my search engine traffic seems to have dropped by around 50% Could this have anything to do with the robots.txt - or just a big co-incidence? For the time being I've removed the robots.txt to see if my traffic levels go back up. This could be completely unrelated, but about a week after I added the above basic robots.txt file, my search engine traffic seems to have dropped by around 50%

Could this have anything to do with the robots.txt – or just a big co-incidence?

For the time being I’ve removed the robots.txt to see if my traffic levels go back up.

]]> By: Everton http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-63595 Everton Mon, 07 May 2007 17:23:39 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-63595 - open notepad - paste the text you want to use - save the file as robots.txt - upload it to your blog's root directory - open notepad
- paste the text you want to use
- save the file as robots.txt
- upload it to your blog’s root directory

]]> By: Everton http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-222060 Everton Mon, 07 May 2007 17:23:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222060 - open notepad - paste the text you want to use - save the file as robots.txt - upload it to your blog's root directory - open notepad
- paste the text you want to use
- save the file as robots.txt
- upload it to your blog’s root directory

]]> By: Vikram http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-63588 Vikram Mon, 07 May 2007 16:25:05 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-63588 can anyone help me on how to install the Robots.txt file ... Please help me Please reply to vicky316[at]gmail[dot]com can anyone help me on how to install the Robots.txt file … Please help me

Please reply to vicky316[at]gmail[dot]com

]]> By: Vikram http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-222059 Vikram Mon, 07 May 2007 16:25:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222059 can anyone help me on how to install the Robots.txt file ... Please help me Please reply to vicky316[at]gmail[dot]com can anyone help me on how to install the Robots.txt file … Please help me

Please reply to vicky316[at]gmail[dot]com

]]> By: Glenn http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-63436 Glenn Sun, 06 May 2007 21:24:22 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-63436 What does the below command do? Disallow: /wp- Does it disallow all directories that start with wp- in the WordPress blog? I too got bit by Google around 5/1/07 and nealy all of my blog posts went into the Supplemental result decreasing my traffic and sales overall. Hope it works. What does the below command do?

Disallow: /wp-

Does it disallow all directories that start with wp- in the WordPress blog?

I too got bit by Google around 5/1/07 and nealy all of my blog posts went into the Supplemental result decreasing my traffic and sales overall.

Hope it works.

]]> By: Glenn http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-222058 Glenn Sun, 06 May 2007 21:24:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222058 What does the below command do? Disallow: /wp- Does it disallow all directories that start with wp- in the WordPress blog? I too got bit by Google around 5/1/07 and nealy all of my blog posts went into the Supplemental result decreasing my traffic and sales overall. Hope it works. What does the below command do?

Disallow: /wp-

Does it disallow all directories that start with wp- in the WordPress blog?

I too got bit by Google around 5/1/07 and nealy all of my blog posts went into the Supplemental result decreasing my traffic and sales overall.

Hope it works.

]]> By: Zath http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-63341 Zath Sat, 05 May 2007 21:13:21 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-63341 I've now added a robots.txt file to my site, it's only basic one for wordpress blogs so far, but I'll see what, if any difference that makes to start with. I’ve now added a robots.txt file to my site, it’s only basic one for wordpress blogs so far, but I’ll see what, if any difference that makes to start with.

]]> By: Zath http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-222057 Zath Sat, 05 May 2007 21:13:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222057 I've now added a robots.txt file to my site, it's only basic one for wordpress blogs so far, but I'll see what, if any difference that makes to start with. I’ve now added a robots.txt file to my site, it’s only basic one for wordpress blogs so far, but I’ll see what, if any difference that makes to start with.

]]> By: Jen http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-62840 Jen Mon, 30 Apr 2007 23:54:03 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-62840 Hm, I haven't added a robots.txt file yet, but I think I will now! Thanks for the idea. Hm, I haven’t added a robots.txt file yet, but I think I will now! Thanks for the idea.

]]> By: Jen http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/comment-page-5/#comment-222056 Jen Mon, 30 Apr 2007 23:54:00 +0000 http://www.connectedinternet.co.uk/2007/04/12/adding-a-robotstxt-file-has-increased-my-google-traffic-by-16-in-4-days/#comment-222056 Hm, I haven't added a robots.txt file yet, but I think I will now! Thanks for the idea. Hm, I haven’t added a robots.txt file yet, but I think I will now! Thanks for the idea.

]]>