1. Jul 12th, 2006

    The UI is the API: Scraping with Ruby

    ui-is-api.png

    Here’s my presentation from Mashup Camp II.

    To view this presentation you’ll need a computer to which you can download, install and run a Web browser.

    Wait … you already have that?

    Isn’t HTML great?

    That’s half the presentation right there. For all the hype around Web services, SOAP and RDF, sometimes its the simple protocols that do the most. So let’s use some HTML/HTTP.

    I start by busting a few of the myths surrounding HTML/HTTP and scraping in general. In real life it works better than you’d expect.

    How come? Because simple solutions are more resilient than we give them credit. And what’s easy to build, it also easy to test and easy to fix.

    The second part shows how to scrape eBay auctions in ten lines of code. There’s also a very brief introduction to Ruby, just enough to understand the examples. If you know Ruby, skip that. If you don’t know Ruby, change that!

    And it ends with metrics from a real live application. You know the one I’m talking about.

    I wrote the framework so I can spend less time writing scrapers, and more timing working on features that matter. So have fun and explore.

    N.B. Someone asked if you could use scrAPI for testing Web UIs. Yes. But you might find assert_select easier to use. It shares the same code and style (pun intended) but geared towards test cases.

    Posted by Assaf Filed in scrapi

    1. Jul 12th, 2006

      Nir

      Good stuff - nice to see how tasks we’ve all been doing with varying degrees of kludging get so cleanly implemented in Ruby ;)

    2. Jul 12th, 2006

      Pradeep Sethi

      I missed your presentation today. This stuff looks great though. Are you doing any sessions tomorrow?

    3. Jul 12th, 2006

      Assaf

      Pradeep,

      Apparently so did I :-(

      I did one on Monday, last minute schedule shuffle, but didn’t get to MashupCamp in time today to do another one.

      And I think the Monday one, just wasn’t good. I didn’t get a lot of time to prepare, I knew that upfront but wanted to do it anyway. I’ll have to polish it up and do it again at another event.

    4. Oct 27th, 2006

      Labnotes » Rounded Corners - 53

      [...] The UI is the API. Drew McLellan asks: can your Web site be your API? Let me ask a different question: why is your Web site not your API? Is there a technical reason for that, or are we just conditioned to think that UIs are pretty and APIs are “complex ugly beasts” the need a lot of SOAP? [...]

    Your comment, here ⇓

    Or using OpenID