1. Jul 21st, 2005

    Who’s reading the blogsphere?

    I’ve been following Robert Scoble’s comparison of the various blogsphere search engines. It was interesting to read what he found out, but I wasn’t prepared for what I found in the blog stats this morning.

    This blog has been up for only a few short days. It shows up on technorati and bloglines, I added it there out of curiosity to see how fast they can track new posts. Technorati is back on track, and was very quick in adding one of my posts to the tags tag. I didn’t bother with the rest, but my blog does ping to ping-o-matic. That’s the default setup for Wordpress.

    Right now, the search engine saturation is shown as 0 (zero, as in nada), meaning that Google, Yahoo and MSN don’t know it exists. Which is fine. But I am seeing spiders grabbing the feeds, from feedster, icerocket, blo.gs, syndic8, blogpulse, blogslive, topicblogs, and
    omni-explorer.

    Now, here’s the interesting bit. The last three are future services. The omni-explorer site has this message:

    Omni-Explorer is a venture-backed startup based in Silicon Valley. Stay tuned to this site; we plan on launching shortly.

    If you have found this page because of the Omni-crawler, please bear with us. We hope to be able to point many more users to the valuable content on your site shortly.

    You’re with me? All these startups are building new services that know what’s going on in the blogsphere. They know about a blog that barely exists. And it’s not rocket science, you just need to listen to the pings. But all of that, and this post as well, goes under the radar of the big three.

    P.S. And another cool use of technology. The last three spiders report their URL in user agent header, so the site name shows in my logs. That’s how I identified them. Leave clues and people will link back to you.

    Update: The Googlebot trails the RSS spiders by three days.

    Update: Interesting entry about omni-explorer.

    tags: blogsphere rss-search

    Posted by Assaf Filed in general 13 Comments »

    1. Jul 22nd, 2005

      Lab notes :: This blog is moving … :: July :: 2005

      [...] And so is the RSS feed, so you might want to switch here. [...]

    2. Jul 22nd, 2005

      Steven Kaye

      Interesting that Omni-Explorer plans on honoring robots.txt files.

    3. Jul 22nd, 2005

      Richard Gendal Brown

      Search Engines

      More evidence that we’re still in the “Creative Destruction” phase…. who will win in the wash out?

    4. Jul 22nd, 2005

      josh

      I’m curious what your experience with the Omni Explorer was. It came through my site and absolutly trashed things. It spent two hours fetching every page I had serially with barely a pause between fetches.

      How did it behave on your site?

    5. Jul 22nd, 2005

      SiliconBeat

      What is Omni-Explorer?

      Some folks are starting to notice a new web crawler sniffing around their web sites. Their web logs point people to site for Omni-Explorer, “a venture-backed startup based in Silicon Valley.” Do we have anohter search company in our midst? (via Sco…

    6. Jul 22nd, 2005

      Geek News Central

      More Weblog Search Sites joining the Party

      I have not really been watching my raw log files for blog specific search engines, at least not until today…

    7. Jul 22nd, 2005

      Omni Guy

      Josh,

      That was the old version of our bot – the new one should be better behaved. If you still have issues, drop us a note at feedback (at) omni-explorer (dot) com. Thanks.

    8. Jul 23rd, 2005

      Don Singleton

      Who’s reading the blogsphere?

      The blogs to be spidered the fastest are Blogger.com (BlogSpot), because that is owned by Google, but it seems to pick up on the others pretty fast.

    9. Jul 25th, 2005

      Assaf

      some posts already show up on Google but not on Yahoo, even though Yahoo shows pages that link to the blog.

      the logs do show googlebot, yahoo/inktomi, yahoo feed seeker and msnbots in the list, along with PubSub and a few others I don’t recognize: SVRBot, TutorGigBot and Larbin.

    10. Jul 27th, 2005

      John Male

      hz2 hz2 hz2 hz2 http://www.getfirefox.com

    11. Sep 14th, 2005

      [blog.forret.com]

      Blog search engine frenzy

      Exciting times in blog search country:

      Technorati seems to have figured out how to make its cosmos search fast again. About time, as I said earlier.

      Google has just launched its long awaited blog search. It is also available under the Blogger bra…

    12. Jan 31st, 2006

      Bob

      I believe I found the source on omni-explorer. You will be shocked.
      Contact me if you are interested.

    13. Mar 15th, 2006

      peter.forret.com » Blog Archive » Blog search engine frenzy

      [...] Labnotes reports on more search engines to surface soon: blogslive, topicblogs, and omni-explorer. [...]