Thursday, Mar 31, 2005

Screen-Scraping with OPS

I read a couple of days ago an article by Brian Goetz on IBM DeveloperWorks about with XQuery.

The idea behind HTML screen-scraping consists in accessing an HTML page on the web, and extracting information out of it. The DeveloperWorks article proposes two ideas:

  1. Using JTidy to convert the HTML to well-formed XML
  2. Using XQuery to extract and reformat the data

I have to admit that I was not familiar with the term "screen-scraping", but in fact we've had some examples of this technique in OPS for a very long time. In particular, the URL Generator example was retrieving the latest CNN headline by accessing the HTML of the CNN web site, and the Google Spell-Checker example was doing even more complex HTML-based interaction with Google.

The URL generator processor in OPS is able to produce XML from data fetched from a URL that you pass it. That source data may be of four different types:

  • XML. In this case, it is just parsed
  • HTML. In this case, it is automatically cleaned-up (with JTidy!) and converted into XML
  • Text. In this case, the text is encapsulated within XML
  • Binary. In this case, the binary is Base64-encoded and encapsulated within XML

So it is trivial to extract an existing HTML page from OPS with code like this in XPL:

  <p:processor name="oxf:url-generator" xmlns:p="http://www.orbeon.com/oxf/pipeline">
  <p:input name="config">
  <config>
  <url>http://weather.yahoo.com/</url>
  <content-type>text/html</content-type>
  </config>
  </p:input>
  <p:output name="data" id="page"/>
  </p:processor>

Now the thing we did not do with the URL generator example was using XQuery: we used XSLT instead. In fact, XQuery and XSLT are very similar in what they can accomplish in such a use case, the XQuery syntax being a little lighter. OPS had an XQuery processor as well, illustrated by the XQuery sandbox example. We were using a fairly old version of Qexo, which implemented a version of the XQuery spec that was not quite up to date. I used the opportunity to move to Saxon. Saxon is primarily known as an XSLT transformer, but because XQuery 1.0 is so close from XPath 2.0 and XSLT 2.0, Saxon also implements XQuery. Since Saxon is already the default XSLT transformer in OPS, implementing an XQuery processor based on it was a breeze. Here is the Yahoo! Weather example from the article, written in XPL:

  <p:config xmlns:p="http://www.orbeon.com/oxf/pipeline">
  <p:processor name="oxf:url-generator">
  <p:input name="config">
  <config>
  <url>http://weather.yahoo.com/</url>
  <content-type>text/html</content-type>
  </config>
  </p:input>
  <p:output name="data" id="page" debug="page"/>
  </p:processor>
  <p:processor name="oxf:xquery">
  <p:input name="config">
  <xquery>
  <html>
  <body>
  <table>
{ for $d in //td[contains(a/small/text(), "New York, NY")] return for $row in $d/parent::tr/parent::table/tr where contains($d/a/small/text()[1], "New York") return
  <tr>
  <td>{data($row/td[1])}</td>
  <td>{data($row/td[2])}</td>
  <td>{$row/td[3]//img}</td>
  </tr>
}
  </table>
  </body>
  </html>
  </xquery>
  </p:input>
  <p:input name="data" href="#page"/>
  <p:output name="data" id="html-page"/>
  </p:processor>
  <p:processor name="oxf:html-serializer">
  <p:input name="config">
  <config/>
  </p:input>
  <p:input name="data" href="#html-page"/>
  </p:processor>
  </p:config>

As usual with XPL, the XML pipeline language of OPS, you notice how easy it is to connect together small components and make them work together without writing any line of Java and without going through compilation and deployment.

Now here is how you could write the XQuery fragment with XSLT:

  <p:processor name="oxf:xslt" xmlns:p="http://www.orbeon.com/oxf/pipeline">
  <p:input name="config">
  <html xsl:version="2.0">
  <body>
  <table>
  <xsl:for-each select="//td[contains(a/small/text(), 'New York, NY')]" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:variable name="d" select="."/>
  <xsl:if test="contains($d/a/small/text()[1], 'New York')">
  <xsl:for-each select="$d/parent::tr/parent::table/tr">
  <xsl:variable name="row" select="."/>
  <tr>
  <td><xsl:value-of select="$row/td[1]"/></td>
  <td><xsl:value-of select="$row/td[2]"/></td>
  <td><xsl:copy-of select="$row/td[3]//img"/></td>
  </tr>
  </xsl:for-each>
  </xsl:if>
  </xsl:for-each>
  </table>
  </body>
  </html>
  </p:input>
  <p:input name="data" href="#page"/>
  <p:output name="data" ref="data"/>
  </p:processor>

This Yahoo! Weather example using XQuery is now in CVS and should have already shown up in the unstable builds.

OPS Screen-Scraping Example
The Yahoo! Weather Screen-Scraping Example in OPS

Posted by Orbeon at 2:48 am UTC