Catching the Googlebot

From Dotfactor.com

Search Engines
Catching the Googlebot
By Jan Hopkins

Catching Google and Googlebot at the Right Time

To have your website included in the Google database, or have your site's updates reflected in the database, as soon as possible, you need to do some planning and preparing so that you can catch Googlebot at the correct position in the cycle. We know that there is the initial Googlebot crawl in the beginning of the month, as well as a crawl during and directly after the update.

So, if you wish to have a new website included in the DB, will either of these crawls insure your inclusion into the database? It has been seen that this is not always the case. Mainly, if your website is crawled in the beginning of the month, the chances are that your website will not be included in that months update. If your website is crawled during the second crawl of the month, which is directly following the update, it is possible that your website will be revisited in the next crawl and then included in the next update. Other times Google will visit a new site and grab only the robots.txt and the homepage. This is a good indication that Googlebot will be back during the next major crawl and your website will be included in the update following that second crawl. So, looking back, it seems that for your new site to be included in the Google database it will take two visits from Googlebot. This is true for most cases.

To get the fastest inclusion time possible, there are a few things that you can do. If your website is crawled for the first time by Googlebot during or directly after the update, then you are in good shape as it is more than likely your website will be included in the next month's update. If your website is not crawled at this point, but during the next crawl, you will have to wait even longer for your website to be included in Google's database. So, how do you get Googlebot to crawl your website during that specific time period? You can either hope that it will happen this way (not very reliable, is it?), or you can do your homework and plan it out. If you have other websites that are in the Google database, you can watch the crawling and update dates and then plan your launch accordingly. If you don't have any websites in the Google database that you can monitor, you can either watch Google.com for the updates, or you can read about crawling and update times at WebMasterWorld.com's Google forum.

While there is almost no way to be 100% sure that your website is going to be crawled, there are certain things that you can do to get Googlebot's attention and attract the robot to your site. The first thing to do is obtain links to your site from other websites with a high PageRank. The higher the PageRank of a website, the more that website will be crawled and refreshed by Google, which means your link will be picked up more quickly. Secondly, you can submit your website to Google through their Add URL page. This is not a definite way into the database, and is not 100% reliable. The third thing that you can do is install the Google Toolbar, and then visit your own website through the toolbar. There have been numerous reports of a direct correlation between a website's inclusion into the database and a visit through the Google Toolbar. While it will cost you $299 annually, a Yahoo! directory listing is also a help in getting into Google's database, and Yahoo! offers quick inclusion times (7 days) into their directory. An Open Directory Project listing is also a very good way to have your website included in the database, although this can sometimes take long periods of time, and is not 100% reliable.