Some Tips For Setting Up Your Sitemap
XML sitemaps are used to help search engine crawlers do a better job of finding and prioritizing all of your content. It doesn’t guarantee that your pages will be indexed, but it does assist the crawlers in determining what content you don’t want indexed, and what content you would like to see a higher priority placed on.
There are several web-based xml sitemap generators, and in particular a very good Wordpress plugin that will do the job for you and make it a plug and forget affair.
The important things to keep in mind when setting up your XML sitemap are 1) figuring out what content you do not want indexed, and 2) figuring out how to include your content so that the pages you really want showing up in the search results do.
There are pages on your site that you do not want to include in your XML sitemap. It’s great having a lot of content to submit to the search engines, but you want search engine results to return content, not your Contact page.
Some of the content you may not want showing up in search results:
- Contact form pages
- About pages
- Poll results pages
- Links pages
- Users online pages
- Category pages
- Archive pages
- Comments pages
- Disclaimers and disclosure policy pages
Keep a watchful eye on your polls, some poll plugins generate duplicate pages! One of the surefire ways to incur Google’s wrath.
What do you want to have the search engines return in search results? Your content or the comments? Content.
Starting to get the idea? Say you’ve targeted the keyword schnobblegrueber. When someone searches schnobblegrueber on Google, you want them to land on search results that send them to articles you have written that are relevant to schnobblegrueber. This is where the category and archive sections can cause a problem if you are submitting them to be indexed.
You want people to enter on:
yoursite.com/category/schnobblegrueber/ways-to-increase-your-sites-schnobblegrueber
as opposed to:
yoursite.com/category/schnobblegrueber/3
or:
yoursite.com/2007/10/12/ways-to-increase-your-sites-schnobblegrueber
The problem with 2 of the 3 above urls?
- yoursite.com/category/schnobblegrueber/3 is constantly changing if you write more articles in that category. You don’t want people to get a search result that takes them to this page on your site, because the schnobblegrueber content may not be there anymore. Now they bounce off to somewhere where they can get what they were looking for, where if your search result would have returned the article page they might have stuck around for some pageviews and ad clicks.
- yoursite.com/2007/10/12/ways-to-increase-your-sites-schnobblegrueber creates duplicate content on the search engines. They don’t like that, and it’s confusing to have two urls for one piece of content.
You probably do want the search engines to display your category pages, but remember that Google won’t only end up displaying the first page in your category listing but /2 /3 /4 ad infinitum as well. Set them to have a lower priority than your individual article entries.
Starting to get the idea? You want the search engines to like you and think you’re one of the places they should send people. 1200 entries in your XML sitemap may look impressive, but often 500 entries that direct the search engines to what you really want people to see is much more effective. Avoid duplicate content, don’t submit items that people aren’t going to be searching for, prioritize how you want the search engines to display your content, and you have just taken another step toward increasing your serps.


Comment by Youpark on 12 December 2007:
Was looking for such information. Thanks for such a comprehensive post!
Comment by Joseph Plazo on 12 December 2007:
I’d like to recommend the SoftPlus Gsite Crawler for whipping up sitemaps. I’ve found it effective for htm, php and other web documents. It’s free and handles unlimited number of pages. Simply install, let it run during idle PC time, then upload the sitemaps.
This baby indexed all 39,000 documents on one of my sites in 4 hours!
Comment by degree on 12 December 2007:
That’s cool software, but if you have no computing power to spare, you can let the free service xml-echo.com create your sitemap for you. You’ll have to queue. After half a day, you’ll be emailed a free sitemap.
Be wary- only 500 documents will be mapped.
Comment by Amanda on 12 December 2007:
I was just looking for an article like this when I noticed your post in my feed reader. Thanks for the info!
Comment by mlankton on 12 December 2007:
Keep in mind that the nice thing about a sitemap plugin is that you don’t need to do anything when you add content. The plugin updates your sitemap every time you publish an item.
Comment by Sergio on 13 December 2007:
what sitemap generator you guys use to generate a sitemap over 500 pages?
i tried xml sitemaps but it only genrates 500 pages max.
Comment by mlankton on 13 December 2007:
What are you using to generate your website? An XML sitemap can contain 50,000 urls, or 10MB in size, whichever comes first. I know that both the Wordpress and Joomla XML sitemap plugins have no such 500 url limit.
The online generators want you to pay for their service if you need a large map. Whatever you are using to manage your site, their is probably some kind of XML sitemap extension for it that will do the job.