Comments on: Creating A WordPress Robots.txt To Improve SEO

By: free avatars

free avatars — Wed, 01 Sep 2010 09:48:04 +0000

I enjoyed reading this, are you on twitter?

By: Golf Clubs

Golf Clubs — Sun, 29 Aug 2010 20:25:54 +0000

Tiger always gives 110 percent. That is why he gave 100 percent to his wife and still had 10 percent left over for his alleged mistress.

By: registry cleaner

registry cleaner — Sat, 21 Aug 2010 10:03:04 +0000

Prompt, where I can find more information on this question?

By: Books wholesale

Books wholesale — Tue, 17 Aug 2010 04:23:59 +0000

Olvasni egy nagyszerű időtöltés, hogy elősegíti a kreatív gondolkodást, és ad egy nagy beszélgetés indításához, de nehéz lehet tudni, hogy hol kezdjem. Reading listák segítségével eltartás ön összpontosító olvasási, és motivál, hogy olvassa el többet, mint amennyit egyébként is. Tudnia kell, hogyan válassza ki könyveket, hogy élvezni fogja, és hozzon létre egy hasznos, kellemes és szórakoztató olvasás listán. Ez a lista olvasó, meg fogja találni azt a hatékony olvasni. Könyvek igazán könyvek, és ők nem a díszítés sajátterem.

By: Best Forex Signals

Best Forex Signals — Sat, 14 Aug 2010 14:53:14 +0000

Whats happening, I found this website by mistake when I had been going through Bing after that I arrived to your web site. I need to say your site is interesting I really like your theme! Today I don’t have the free of charge time at the existing moment to fully look through your sitebut I have bookmarked it. I is going to be back again in a day or 2 . Thanks for a great website.

By: Forex Trading Alert

Forex Trading Alert — Sat, 14 Aug 2010 08:53:42 +0000

You may have not intended to do so, but I think you’ve managed to express the state of mind that a lot of individuals are in. The sense of wanting to help, but not knowing how or where, is something a lot of us are going via.

By: registry cleaner reviews

registry cleaner reviews — Thu, 05 Aug 2010 00:24:25 +0000

Thanks for this useful article.

By: learningquranonline

learningquranonline — Mon, 28 Dec 2009 14:38:03 +0000

if any body could guide me that well how can i up load this robot text to my word press blog http://learnquranonline1.wordpress.com

By: Keresőoptimalizálás Könyv

Keresőoptimalizálás Könyv — Sun, 18 Jan 2009 10:10:09 +0000

I think the following site describes perfectly the seo adjustments for a wordpress blog’s robots.txt:
http://codex.wordpress.org/Search_Engine_Optimization_for_Wordpress

By: Keresőoptimalizálás Könyv

Keresőoptimalizálás Könyv — Sun, 18 Jan 2009 10:10:00 +0000

I think the following site describes perfectly the seo adjustments for a wordpress blog’s robots.txt:
http://codex.wordpress.org/Search_Engine_Optimization_for_Wordpress

By: George Donnelly

George Donnelly — Wed, 16 Apr 2008 15:41:04 +0000

What about to block search results?

By: George Donnelly

George Donnelly — Wed, 16 Apr 2008 15:41:00 +0000

What about to block search results?

By: LiveJasmin

LiveJasmin — Sun, 30 Dec 2007 18:42:49 +0000

should this be ok ?

User-agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /cgi-bin/

User-agent: Googlebot-Image
Disallow:
Allow: /*

By: LiveJasmin

LiveJasmin — Sun, 30 Dec 2007 18:42:00 +0000

should this be ok ?

User-agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /cgi-bin/

User-agent: Googlebot-Image
Disallow:
Allow: /*

By: AskApache

AskApache — Tue, 06 Nov 2007 09:04:41 +0000

I finally got my site indexed near perfectly thanks to much reading and viewing my access logs.. Based mostly on this wordpress robots.txt example

By: AskApache

AskApache — Tue, 06 Nov 2007 09:04:00 +0000

I finally got my site indexed near perfectly thanks to much reading and viewing my access logs.. Based mostly on this wordpress robots.txt example

By: stacey

stacey — Thu, 25 Oct 2007 01:11:46 +0000

I fixed the robots.txt. I wasnt putting it in the root directory.

By: stacey

stacey — Thu, 25 Oct 2007 01:11:00 +0000

I fixed the robots.txt. I wasnt putting it in the root directory.

By: stacey

stacey — Tue, 23 Oct 2007 20:00:37 +0000

Hi Everett,

I installed the KB robots.txt plugin. This what I entered in the robots.txt plugin window.
User-agent: *

Disallow:
But when I do I do http://www.babygeartoday.com/robots.txt this is what I get, and in the plugin when it says check the the robots.txt file after I submit I get this also:
# BEGIN XML-SITEMAP-PLUGIN
Sitemap: http://www.babygeartoday.com/sitemap.xml.gz
# END XML-SITEMAP-PLUGIN

So uninstalled the google sitemap and analytics, and still the same thing. I was wondering could you help me solve this problem.

Thanks,
Stacey

By: stacey

stacey — Tue, 23 Oct 2007 20:00:00 +0000

Hi Everett,

I installed the KB robots.txt plugin. This what I entered in the robots.txt plugin window.
User-agent: *

So uninstalled the google sitemap and analytics, and still the same thing. I was wondering could you help me solve this problem.

Thanks,
Stacey

By: Bali Web Design

Bali Web Design — Sun, 14 Oct 2007 13:47:24 +0000

hi, thanks for posting this.
i use it all unkess :
Disallow: /page/ Disallow: /date/ Disallow: /comments/

By: Bali Web Design

Bali Web Design — Sun, 14 Oct 2007 13:47:00 +0000

hi, thanks for posting this.
i use it all unkess :
Disallow: /page/ Disallow: /date/ Disallow: /comments/

By: AskApache

AskApache — Thu, 11 Oct 2007 01:19:31 +0000

Have you read the new robots.txt recently updated article yet? and check out askapache.com/robots.txt

By: AskApache

AskApache — Thu, 11 Oct 2007 01:19:00 +0000

Have you read the new robots.txt recently updated article yet? and check out askapache.com/robots.txt

By: Everton

Everton — Wed, 10 Oct 2007 19:07:43 +0000

Thx Sue for reminding me I need to check my file

By: Everton

Everton — Wed, 10 Oct 2007 19:07:00 +0000

Thx Sue for reminding me I need to check my file

By: Sue

Sue — Wed, 10 Oct 2007 19:00:58 +0000

FWIW: this tester shows a lot of errors:

http://www.searchenginepromotionhelp.com/m/robots-text-tester/robots-checker.php

Sue

By: Sue

Sue — Wed, 10 Oct 2007 19:00:00 +0000

FWIW: this tester shows a lot of errors:

http://www.searchenginepromotionhelp.com/m/robots-text-tester/robots-checker.php

Sue

By: Paula Mooney

Paula Mooney — Tue, 09 Oct 2007 16:38:13 +0000

Thank you so much for this post.

I found out I had my robots.txt file in the wrong place anyway!

Paula

By: Paula Mooney

Paula Mooney — Tue, 09 Oct 2007 16:38:00 +0000

Thank you so much for this post.

I found out I had my robots.txt file in the wrong place anyway!

Paula

By: Ask Apache

Ask Apache — Thu, 09 Aug 2007 13:18:48 +0000

Hey Everton just stopped by to let you know I created a simpler and updated robots.txt file for WordPress, and I used a lot of the recommendations from your visitor comments.

By: Ask Apache

Ask Apache — Thu, 09 Aug 2007 13:18:00 +0000

Hey Everton just stopped by to let you know I created a simpler and updated robots.txt file for WordPress, and I used a lot of the recommendations from your visitor comments.

By: Ramil

Ramil — Mon, 06 Aug 2007 04:50:27 +0000

Mike has a good point about duplicate contents

By: Ramil

Ramil — Mon, 06 Aug 2007 04:50:00 +0000

Mike has a good point about duplicate contents

By: Zath

Zath — Thu, 12 Apr 2007 19:20:29 +0000

Another area that I’ve never looked into before, I’ll have to read into this and enlightenment myself, but the idea seems like a sound one to me.

I’m guessing it’s best to start with the defaults shown in the linked file on Daily Blog Tips and go from there adding misc pages I have such as pages full of bookmarks etc.

Cheers!

By: Daze

Daze — Thu, 12 Apr 2007 11:19:59 +0000

Good idea to disallow the bots from viewing the RSS feeds. My feeds usually always appear above the actual content in the SERPs which is good, but not so great for humans! Thanks, I’ll give this a go

By: Everton

Everton — Fri, 06 Apr 2007 14:58:43 +0000

if you want your site indexed, why should you exclude something?

Because you don’t want the same posts appearing twice and Google thinking it’s duplicate content, or junk appearing in results as Google will then downgrade your real pages

By: hannes

hannes — Fri, 06 Apr 2007 14:20:45 +0000

if you want your site indexed, why should you exclude something?

By: Mr. Apache

Mr. Apache — Mon, 26 Mar 2007 22:42:41 +0000

There is a better article about this on the askapache blog: WordPress robots.txt optimized for SEO

By: IndoDX

IndoDX — Mon, 26 Mar 2007 21:38:22 +0000

Why Disallow all extension? such as .PHP, .JS? can I get explain more about it?

Thanks

By: Mike

Mike — Mon, 26 Mar 2007 12:02:44 +0000

One of the reasons why you’d want to use a robots.txt file is to prevent Google from indexing content twice and potentially marking it as duplicate content. If it’s listed in monthly archives, category folders and on your front page, for example, you might end up with pages in the supplemental index.

By: SoccerSpeech

SoccerSpeech — Mon, 26 Mar 2007 10:54:00 +0000

I don’t understand Thilak

he wants them to be redirected towards essential pages

What this has to do with the robots file?!
And why should it be a waste of time to crawl the feeds?

By: Thilak

Thilak — Mon, 26 Mar 2007 04:56:29 +0000

I agree with Daniel, but Everton is right. He doesn’t want Googlebots to waste their time spidering unwanted images or feeds. Instead, he wants them to be redirected towards essential pages

By: SoKoOLz

SoKoOLz — Mon, 26 Mar 2007 03:35:59 +0000

omg, this is very useful.
thx u thx u

By: bill

bill — Sun, 25 Mar 2007 19:45:00 +0000

Everton, I’d recommend using a mixture of robots.txt and meta tags. For things like your archives and tag directories you’d want to have a meta tag which allowed following but not indexing. This prevents you from getting your archives entered into Google but allows the posts to still get crawled.

Putting meta name=”robots” content=”noindex,follow” at the top of your archive and tag templates will allow the links to be followed but should prevent the actual archive page from being indexed.

In your robots.txt you should probably only disallow anything that you want a well behaved robot to completely ignore. If files or directories aren’t specifically linked to (and your wp-includes, wp-admin, and wp-content directories are among these) then you can leave them out altogether.

A well behaved spider will only follow links. Any spider, well behaved or not, won’t go anywhere that isn’t specifically referred too. If there’s no link to a directory it doesn’t know that directory exists.

By: SoccerSpeech

SoccerSpeech — Sun, 25 Mar 2007 16:56:00 +0000

Well Everton, I`m not so good in this but I don’t think that you need all of this
For a blog directory, disallowing the SE-bots from visiting the main three folder(wp- admin,-content, -includes) and the wp-”files” in the main directory, will be sufficient! as all .php, .css, .js..etc files will be included in these directories
So a robots.txt file like Daniel or mine will be good.

Anyway, waiting for more comments or asking an expert is the best way to minimize your robots file!

By: Audio Books Fan

Audio Books Fan — Sun, 25 Mar 2007 16:45:00 +0000

I’d leave the robots.txt file as uncluttered as possible. If you make the slightest mistake, the good bots will stop visiting parts of your site. The bots you don’t want o the page will anyhow not pay attention to the robots.txt file.
The example at http://www.dailyblogtips.com/robots.txt is good, you don’t need more than that.

By: Everton

Everton — Sun, 25 Mar 2007 15:54:00 +0000

guys – as I said I’m entering uncharted waters, hence the open question. the example i’ve listed is the biggest one i found.

rather than saying the list is ‘too big’ it’d be more useful if you said what should be included, or what shouldn’t be included

By: Leftblank

Leftblank — Sun, 25 Mar 2007 15:47:00 +0000

I agree with Daniel; why would you want to block that much from Google? Having Google spider your monthly archives will rather have a positive impact as it’ll be able to find your posts.

@Daniel; I don’t see why you’d want to block your /feed/ page from Google though; Google is doing more and more with blog posts, RSS feed parsing is one of these things.

By: Daniel

Daniel — Sun, 25 Mar 2007 13:46:00 +0000

i dont like this version of the robots.txt

why would you want to exclude the monthly archives, for instance?

plus there are some non clear attributes that might end up messing your indexation.

I would rather stick to a more simple and clear robots.txt, you can see then one i am using http://www.dailyblogtips.com/robots.txt