Google has three well known bots: The Adsense bot, the Freshbot and the DeepCrawl.The Adsense bot, as you could probably guess, is used for publishers who have Adsense on their sites. As soon as a new page is created, the JavaScript within the Adsense code sends a message to the Adsense bot, and it will come within 15 minutes to index the page so that it can serve up the most relevant ads.But, for this conversation we are only concerned about the DeepCrawl and the Freshbot.

The Freshbot crawls the most popular pages on your website. It doesn't matter if that is one page or thousands. Sites like Amazon.com and CNN.com have pages that are crawled every ten minutes, since Google has learned that those pages have that amount of frequent changes. A typical site should expect to have a freshbot visit every 1 to 14 days, depending on how popular those pages are.What happens to your site on a Freshbot visit is that it finds all of the deeper links in your site. It places those links into a database so that when the DeepCrawl occurs, it has a reference