Crawl Rate Demystified
According to Google’s Matt Cutts in a video that every blogger should watch, Google crawls according to page rank, or in other words, because it helps determine crawl rate, page rank is still important.
The consensus of opinion among SEO thought leaders is that the higher your Google page rank, the faster Google spiders and indexes your site. If crawl rate tells us how frequently Google spiders your site and is partly determined by how often new pages are added, the faster your crawl rate, the more trust Google has in your content, so the higher you rank.
You can access your crawl stats via Google Webmaster Tools, or now you can check out your crawl rate by adding your site to the Seometer database. Top level domains, for example, http://www.yoursite.com, are free, while there is a $20.00 per year charge for internal pages, such as http://www.yoursite.com/services/. Sub domains such as http://www.services.yoursite.com are also free. So apparently blog urls, such as http://www.yoursite.com/blog are not free. If your blog is a Wordpress blog, however, you can still track your crawl rate free with the Wordpress Crawl Rate Tracker plugin from Blogstorm.
According to the Seometer blog, the average crawl cycle for all sites registered with them is 2.19. So on average Google is updating its cache for these sites once every other day. Comparing your own stat against this average will give you a rough idea where your website is ranked in terms of Google’s crawling frequency.
If you were able to get your hands on a list of directories, blogs, and forums with the fastest crawl rates, you would have a basis for determining which directories to submit to and which blogs and forums to join and comment on. A short list of the top ten most frequently crawled by Google sites on the Seometer home page yields more than a few surprises. For example, who would have thought that the Romow Business Directory (1) would be crawled more often than the Yahoo Directory 1.9)? The top SEO blog with the best crawl rate on the Seometer list is Web Pro News (1.86) and the forum with the best crawl cycle is Web Pro World (1.89). Unfortunately the list only compares sites that are registered with Seometer. Even so, it should still have some value for comparing one site on the list to another.
And finally one way to increase the crawl rate for your inside pages is to build deep links. While most links go to your home page, a deep link is obviously a link that targets one of your inside pages. Here’s a list of free directories by page rank that accept deep links that we submit to as part of our deep link submission service.
