Google, Yahoo and MSN announced last week they are partnering to support a new Canonical URL tag created to help reduce duplicate content issues. As it stands, when search engines discover URLs with essentially the same content they group them into a cluster and select the “best” URL to represent that cluster in the search engine results. But by adding this new HTML tag to a web page you are indentifying it as your preferred URL to the search engines and the version you would like indexed.
Why is duplicate content even an issue? From a user’s standpoint, duplicate content is a nuisance. Google tries to filter out duplicate documents so users have a good, diverse selection of information relevant to their search query instead of a poor selection of duplicate articles or even slight variations of the same article.
Having duplicate content can be a headache for a web site owner too. Certain types of duplicate content can have a negative effect on your site’s performance in the search engines. Oftentimes multiple versions of the same URL are created accidentally by category URL systems, session IDs and other basic site development issues, as well as some marketing tactics. See Google Webmaster Central to learn more about duplicate content caused by URL parameters.
In many cases this new tag is the best way to communicate your preferred URL, but a permanent redirect is still necessary if the duplicate content is on a completely different domain, or if you need to transfer page history or consolidate link credit. Check out SEOmoz for more technical details on how the new Canonical URL tag works.