Fierce internet search rivals Google, Microsoft and Yahoo have partnered to simplify the way webmasters and online publishers submit their sites' content for indexing in the companies' search engines.
In a rare collaborative effort, Google, Microsoft and Yahoo, which compete directly in internet search and other online services, plan to announce today their support for the open-source Sitemap Protocol.
This protocol, based on XML (Extensible Markup Language), was created by Google and has been in use by the company for the past 18 months. Yahoo's adoption of Google's protocol will be effective as of today, while Microsoft plans to aid Google and Yahoo to extend and enhance it. Yahoo will continue to support its existing protocol, in conjunction with Google's, but Microsoft will completely replace its setup with Sitemap Protocol in early 2007.
A site map is a file that webmasters and publishers put on their sites to guide the search engines' automated web crawlers in properly indexing their web pages.
Site maps are useful in highlighting to crawlers the dynamic web content that is served up on the fly. Crawlers generally index content contained in static web pages without problems, but often have difficulty with dynamic content – such as the content that is generated as a result of a search query.
A site map can be formatted using various protocols, but this means more work for webmasters and publishers, which is why Google, Microsoft and Yahoo are throwing their weight behind the Sitemap Protocol to promote it as a standard.
"The benefit for publishers is that they'll get more of their content indexed more rapidly," said Tim Mayer, Yahoo's senior director of global search.
Google, Microsoft and Yahoo believe the common protocol will improve site maps in general, and subsequently make their search engine crawlers more comprehensive in their indexing. This benefit will trickle down to end-users.
"Ultimately what we care about is the best results for searchers and making things easy for site owners. This really does that," said Vanessa Fox, product manager for Google's Webmaster Central.
In addition to listing web pages available for indexing, the Sitemap Protocol lets publishers and webmasters include other relevant information, such as when a page was last updated, how frequently it changes and what its importance level is on the site. All of this leads to more precise and effective crawling.
Google, Microsoft and Yahoo will encourage other search engine operators, as well as makers of related software, such as content management systems vendors, to support the protocol, they said.