How Much Bandwidth Do Search Engines Take Up?

There are an awful lot of search engines out there and they all try to index as much of the web as they can, as quickly as they can.  For this site, search engines seem to cause more traffic than anything else:

<th align="center" bgcolor="#008040" colspan="2">
  <font size="-1">Hits</font>
</th>

<th align="center" bgcolor="#00E0FF">
  <font size="-1">User Agent</font>
</th>

<td align="right">
  <b><font size="-1">6866</font></b>
</td>

<td align="right">
  <font size="-2">12.63%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">msnbot/1.0 (+http://search.msn.com/msnbot.htm)</font>
</td>

<td align="right">
  <b><font size="-1">5600</font></b>
</td>

<td align="right">
  <font size="-2">10.30%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Googlebot/2.1 (+http://www.google.com/bot.html)</font>
</td>

<td align="right">
  <b><font size="-1">3546</font></b>
</td>

<td align="right">
  <font size="-2">6.52%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/</font>
</td>

<td align="right">
  <b><font size="-1">2633</font></b>
</td>

<td align="right">
  <font size="-2">4.84%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com</font>
</td>

<td align="right">
  <b><font size="-1">1706</font></b>
</td>

<td align="right">
  <font size="-2">3.14%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">sna-0.0.1 mikemuzio@msn.com</font>
</td>

<td align="right">
  <b><font size="-1">1120</font></b>
</td>

<td align="right">
  <font size="-2">2.06%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Mozilla/5.0 (compatible; BecomeBot/2.3; MSIE 6.0 compatible;</font>
</td>

<td align="right">
  <b><font size="-1">1103</font></b>
</td>

<td align="right">
  <font size="-2">2.03%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">NetNewsWire/2.0 (Mac OS X; http://ranchero.com/netnewswire/)</font>
</td>

<td align="right">
  <b><font size="-1">806</font></b>
</td>

<td align="right">
  <font size="-2">1.48%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Krell-GeoScraper/0.1 libwww-perl/5.79</font>
</td>

<td align="right">
  <b><font size="-1">754</font></b>
</td>

<td align="right">
  <font size="-2">1.39%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com)</font>
</td>

<td align="right">
  <b><font size="-1">750</font></b>
</td>

<td align="right">
  <font size="-2">1.38%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)</font>
</td>

<td align="right">
  <b><font size="-1">746</font></b>
</td>

<td align="right">
  <font size="-2">1.37%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Planet HUMBUG +http://planet.humbug.org.au/ Planet/1.0~pre1 +</font>
</td>

<td align="right">
  <b><font size="-1">745</font></b>
</td>

<td align="right">
  <font size="-2">1.37%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Planet Linux Australia http://planet.linux.org.au Planet/0.2</font>
</td>

<td align="right">
  <b><font size="-1">744</font></b>
</td>

<td align="right">
  <font size="-2">1.37%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET</font>
</td>

<td align="right">
  <b><font size="-1">726</font></b>
</td>

<td align="right">
  <font size="-2">1.34%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Planet Apache +Unconfigured Planet Planet/1.0~pre1 +http://ww</font>
</td>

<td align="right">
  <b><font size="-1">702</font></b>
</td>

<td align="right">
  <font size="-2">1.29%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">NewsGatorOnline/2.0 (http://www.newsgator.com; 1 subscribers)</font>
</td>

<td align="right">
  <b><font size="-1">673</font></b>
</td>

<td align="right">
  <font size="-2">1.24%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Mozilla/4.0 compatible ZyBorg/1.0 (wn-14.zyborg@looksmart.net</font>
</td>

<td align="right">
  <b><font size="-1">655</font></b>
</td>

<td align="right">
  <font size="-2">1.21%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">NetNewsWire/2.0b45 (Mac OS X; http://ranchero.com/netnewswire</font>
</td>

<td align="right">
  <b><font size="-1">648</font></b>
</td>

<td align="right">
  <font size="-2">1.19%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gec</font>
</td>

<td align="right">
  <b><font size="-1">621</font></b>
</td>

<td align="right">
  <font size="-2">1.14%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">NewsGator/2.0 (http://www.newsgator.com; Microsoft Windows NT</font>
</td>

<td align="right">
  <b><font size="-1">601</font></b>
</td>

<td align="right">
  <font size="-2">1.11%</font>
</td>

<td align="left" nowrap="nowrap">
  <font size="-1">Mozilla/5.0 (compatible; BecomeBot/1.86; MSIE 6.0 compatible;</font>
</td>

Now admittedly a lot of those hits will result in Not Modified responses but still, when you expand this to every site on the internet, that’s a lot of HTTP requests being fired around.

It also shows the power of syndication – almost no one actually reads this site directly (except the bots), yet lots of people link to, comment on or mention my posts (I’m certainly no a, b or even c list blogger though).  Keeps the bandwidth requirements down and still get the message out – can’t complain about that.