XForms Everywhere

With XForms 1.0, xforms:output was limited to displaying plain text. With the introduction of XForms 1.1, xforms:output adds powerful capabilities which include handling images, HTML content and more, depending on your XForms implementation.

To achieve this, XForms 1.1 introduces a new mediatype attribute. You use it to specify the type of the content to display, using the familiar Internet media type (or “MIME type”) syntax. This includes values such as image/* (any image content) or text/html (HTML content).

Orbeon Forms already supported the text/html mediatype, so we had to add support for image mediatypes. To achieve this, we made Orbeon Forms a little smarter.

• Consider that XForms allows you to refer to images not only as URLs (e.g. http://example.org/my/image.png), but as binary data embedded in an XForms instance. With Orbeon Forms, this data is handled on the server. So somehow, this information has to be brought to the client. In addition, browsers typically do not support providing image content as binary (although with some clients this is possible with the data: URI scheme)

• Even when using URLs, consider the following scenario: an image resides on your client computer, you upload it to the server through xforms:upload, at which point the image is available, in your XForms instance, as a temporary server-side URL. Now bind an xforms:output to that image URL: clearly, your browser cannot access that URL directly since it is server-side and therefore private.

Orbeon Forms solves these problems by creating proxy URLs for dynamically produced image content. Concretely, you see in the resulting HTML page paths of the form /xforms-server/dynamic/7FF0AD061DD9BAD27BCA1FF4674C75D6. Orbeon Forms takes care of handling these URLs as needed: you don’t need to worry a bit about this.

That was for the “smart” bit, so how does it look in the end to you, the form author? It’s as simple as this:

<xforms:upload ref="my-image">
  <xforms:label>Logo</xforms:label>
  <xforms:hint>Please upload an image</xforms:hint>
</xforms:upload>
<xforms:output mediatype="image/*" ref="my-image"/>

Yes that’s right: the few lines above handle uploading an image, attaching it to your data, and displaying the uploaded image to the user. Beat that.

We are glad to announce that we will be present at JavaOne 2008 on the OW2 Consortium stand. This year, JavaOne will take place May 6-9.

We have some great Orbeon Forms demos in the works so be sure to stop by!

Wow, it’s been a month since our last post in this blog, and one of the reasons for this is that we have been very busy enhancing Orbeon Forms! Here is a quick list of the most notable additions:

• XPath variables. XPath variables can be used within controls, models and actions, with the xxforms:variable (or exforms:variable) element. Variables are extremely useful, for example to avoid repeating long XPath expressions, or to give an XPath expression unambiguous access to data computed in enclosing xforms:group or xforms:repeat elements. If you are familiar with XSLT variables, you will feel right at home. And once you start using them, you won’t be able to go back!

• XML Schema validation improvements. Inline XML Schema support is now implemented. This allows you to embed XML Schema directly within the xforms:model element. This is particularly useful when you create a new schema which is just used within an XForms document, or to define a series of simple types used for validation. We have also added support for the xsi:type attribute in XForms instances. This is a convenient shortcut which allows you to avoid using binds just to specify types.

• Attribute Value Templates (AVTs) on XHTML elements. In Orbeon Forms, AVTs were already supported on some XForms elements, but you can now use AVTs on XHTML elements as well. This allows you to use XForms to influence aspects of XHTML such as class, style or colspan attributes, and all this without script.

• Action context enhancements. We added the exforms:iterate attribute, XForms 1.1 context() function, and a generalized context attribute on XForms elements (which may appear in XForms 1.2).

• Events enhancements. We now have full support for ev:observer and ev:target attributes on action handlers. We also added support for lists of event names, observers, and targets (an extension which we hope will be available in XML Events 2, currently in Working Draft at W3C). Finally, you can now pass context information to custom events with the extension xxforms:contex child element of actions such as xforms:dispatch or xforms:event.

• Image handling. xforms:output supports displaying images when used with xs:anyURI or xs:base64Binary datatypes and an image mediatype, as per XForms 1.1. It has never been easier to upload and display images from XForms!

• XPath extension functions. We added XPath functions to access headers and request parameters directly from XForms (xxforms:get-request-parameter() and xxforms:get-request-header()), as well as extension functions to create elements and attributes by name (xxforms:element() and xxforms:attribute()).

• Submissions. Submissions allow PUT and POST of binary content directly from XForms. This makes it very easy to upload a binary file using xforms:upload and then send it out to a service through xforms:submission.

For more information, check the temporary list of changes for the upcoming Orbeon Forms 3.7! We already have over one hundred enhancements implemented since version 3.6.

XForms 1.1’s small number of actions are surprisingly powerful. By combining them, you can do almost any operation you can think of within an XForms page. Yeah, XForms is Turing complete.

There are however a few use cases that remain difficult to achieve. For example, you can easily repeat an action while a condition is satisfied with the while attribute. However, you cannot yet directly iterate over a sequence of node and perform a group of actions for each iteration. As the XForms Working Group is working on fixing this, we have implemented in Orbeon Forms the exforms:iterate attribute (also available as xxforms:iterate attribute) on XForms action elements. This attribute was first proposed as part of the eXforms extensions.

So how does it work? Consider the following three XForms instances:

    <xforms:instance id="main-instance">
        <instance/>
    </xforms:instance>
	
    <xforms:instance id="source-instance">
        <instance>
            <title>Don Quixote de la Mancha</title>
            <author>Miguel de Cervantes Saavedra</author>
            <title>Jacques le fataliste et son maître</title>
            <author>Denis Diderot</author>
            <title>Childhood's End</title>
            <author>Arthur C. Clarke</author>
        </instance>
    </xforms:instance>
	
    <xforms:instance id="template-instance">
        <book>
            <title />
            <author />
        </book>
    </xforms:instance>

The goal is to take data from source-instance, and copy it over to main-instance but formatted differently. Here is a way to do it:

    <xforms:action xxforms:iterate="instance('source-instance')/title">
        <xforms:insert context="instance('main-instance')" nodeset="book" origin="instance('template-instance')"/>
        <xforms:setvalue ref="instance('main-instance')/book[last()]/title" value="context()"/>
        <xforms:setvalue ref="instance('main-instance')/book[last()]/author" value="context()/following-sibling::author"/>
    </xforms:action>

The outermost action iterates over the <title> elements of source-instance. For each of those, a new <book> element, copied from the template stored in template-instance, is inserted into main-instance. Then values from the <title> and <author> elements are copied over to the new structure. The XForms 1.1 context() function, also recently implemented in Orbeon Forms, provides access to each of the iterated nodes. The resulting main-instance is as follows:

    <instance>
        <book>
            <title>Don Quixote de la Mancha</title>
            <author>Miguel de Cervantes Saavedra</author>
        </book>
        <book>
            <title>Jacques le fataliste et son maître</title>
            <author>Denis Diderot</author>
        </book>
        <book>
            <title>Childhood's End</title>
            <author>Arthur C. Clarke</author>
        </book>
    </instance>

It looks easy this way, doesn’t it?

A couple of weeks ago I attempted to order prints from Flickr. I added a few photos to my cart and was redirected to a service called QOOP. This has a neat interface that allows you organize photos for printing in a book format. I played a bit with the interface and then was caught by other tasks. When I came back to my browser window and attempted to preview the photo album, I got a wonderful error message about a session having expired. The result: my album was lost. I gave up.

My adventure with expired sessions is quite common and I can bet you have had this kind of issues in the past. The least that can be said about this is that it is not user-friendly!

To address this kind of issues, we recently introduced a new feature in Orbeon Forms called the “session heartbeat”. The idea is that if you happen to leave a browser window open on your computer, chances are that you will get back to that window and keep using the application. The last thing you want to happen when you come back is lose your session and therefore your data, as happened to me.

This is not always a correct guess of course: you may just happen to leave a window or tab open without planning to use it again. Conversely you may have a page which is not actually visible, for example in your browser history, yet you will come back to it. This approach wouldn’t be good for banking applications either. Still, in many situations, such as filling-out large forms, it sounds like a good idea to keep your session alive for open pages.

To achieve this goal you could make all server sessions longer. However this is harder to configure for the application developer, and this won’t discriminate between pages that are actually open on a client and the ones that are not. And while it may be ideal to have infinitely long sessions, unfortunately many applications are not ready for this kind of approach.

So we turned to the idea of implementing a “session heartbeat”. Here is how this works:

• When this feature is enabled, an open XForms page regularly pings the server through Ajax to keep the current session alive.

• The ping delay is automatically computed based on the server’s session timeout. The client pings the server at 80% of the session expiration time after the last interaction with the server.

• We are careful not to hit the XForms engine too much, in fact we do a very limited amount of work on the server for each ping, so they should run fast.

• XForms state information for pages hit with the heartbeat just migrates to the disk store over time if RAM is used by other pages, so keeping even large numbers of pages open should not have any negative impact on server RAM.

• When a user gets back to using the page, state information migrates back from disk to RAM, and the page will be live again.

• Sessions do eventually expire as nobody keeps a browser open forever.

Note that whenever an application keeps sessions alive for a long time, it is a good idea to keep as little data as possible in the session. The Orbeon Forms XForms engine itself uses a global state store and does not use session objects for storage, but do remember to keep your sessions small!

Hopefully, this will help prevent many occurrences of the infamous “session expired” error message. The beauty of it is that as an Orbeon Forms application developer you don’t have to worry about anything: the session heartbeat is enabled by default (but it can be turned off globally in properties.xml or individually for each page).

I recently booked a flight on the Continental web site. I knew approximately the flight I wanted to book, so dug right away into the search to select it. I encountered the following issues:

• When the results came, the prices were in Swiss francs. Since I was in Switzerland but using a US credit card, I decided to switch my location using the quite nicely done language and location menu available on the result page.

  I first noticed a funny glitch: for Switzerland, the language selection was English and German. English was selected by default, which suited me, but English was the second radio button in the list. When I switched to the United States location, the second radio button remained selected. However, since for that location the language selection was English first, then Spanish, the result was that the Spanish language was selected. Oops.

• This was just a little bug though. The bigger problem was that as the language was switched, somehow the search page kept my flight dates, but lost the selected flights. So I had to go through them again.

• When doing the final checkout, I omitted a digit in the credit card number. It took the web site almost one minute, after I pressed the submit button, to tell me that there was a problem with that number.

• When that was fixed, I also happened to have forgotten to enter the billing address. The same thing happened: I had to wait for quite a long time after submitting the form to get a message telling me that that was wrong.

Now let’s say the developers of the Continental web site decide to switch to Orbeon Forms and XForms for the next version of the web site:

• With XForms, they easily set constraints, such as saying that the value of the radio button should be the currently selected value (say, ‘en’) if available, or then the default value if unavailable.

• Localized user interface resources are stored in a separate XForms instance (which is an XML document). When switching the resources document, the entire user interface updates without losing the other data captured by the user.

• XForms has several mechanisms to check input data: you can use XML Schema types or XPath constraints (XForms 1.1 even supports a credit card number type) to make sure that a Visa or Mastercard number is at least approximately correct in length and that the redundancy digit applies correctly.

• XForms also support ways to mark fields as required so the user doesn’t forget to enter some information.

And this is all interactive too, meaning that the user doesn’t need to wait one minute to know that some fields are missing or incorrect.

We had great success with the XForms Evening a the XML 2007 conference on Monday. We were wondering if people would really want to attend sessions that late in the evening (from 7:30 to 9:00 PM), but the attendance beat our expectations, with easily over 60 people cramped in the allocated room and some actually standing.

The initial feedback we are getting is excellent. I believe that the format of the sessions was one of the reasons for this: it consisted in having a one and a half hour session on a specific topic (XForms) but split into a series of very short 15-minute presentations by different speakers. This made it almost impossible to bore the audience, and also constrained the speakers to be concise and to the point. So kudos to the organizers, in particular to our working group chair, John Boyer, for making this happen.

The main point of my presentation was that with XForms and eXist, you don’t need a middle tier for a wide category of applications: XForms can directly talk to the persistence layer, in this case eXist, with xforms:submission and REST, while providing all the CRUD and search operations (through XQuery). This point resonated with other speakers’ point of view as well.

Elliotte Rusty Harold ended the evening with an excellently delivered but provocative keynote about what XForms proponents need to do to ensure the sucess of XForms. At Orbeon, we disagree with Elliotte’s contention that it is necessary to bring XForms natively to the web browser:

• After all, Java, PHP and Ruby are a success today on the web without being delivered to the browser.

• XForms implemented as a combination of server-side and client-side software in fact has considerable benefits, such as ensuring the security of the data handled by your application, ease of upgrade, and especially the fact that this allows to deliver XForms today to mainstream web browsers.

• Finally, XForms as a forms technology (as opposed to a general-purpose UI technology) fills a huge gap in the industry. This, along with quality implementations (among which we hope Orbeon Forms stands) able to deliver XForms to the web, will secure its future.

You can read the Forms Working Group’s short debriefing in our meeting notes for today, and the slides of my talk are available on slideshare.

Yesterday, I was filling-out an application form on the E*TRADE web site.

So here I go, entering my information in a traditional wizard-like workflow, with page reloads et al. When I get to the end of it (about six or seven pages total), I figure I would check if I could amend my mailing address on page two. So I carefully press the “Previous” button of the wizard, until I reach the page. So far so good, my information is still there (even though it turns out I can’t change this particular bit). This is the beginning of the trouble:

• First issue: there is no way to complete the workflow from page two, even though I have completed all parts of the wizard. I have to press the “Next” button again and go through the remaining pages all the way to the end.

• Second issue: as I am doing so, part of the information I had just entered is not restored at all. It had displayed just fine as I was navigating back in the wizard, but now that I am navigating forward, it is missing and I have to re-enter it!

• Third issue: finally getting to the end of the wizard again, I complete my workflow and reach a summary page. That page has buttons to edit particular parts of the information. I click on one of those as a last attempt to change my mailing address. This takes me to page two again, just to find out that no, I really can’t change it. This time, there is no way to just go back to the summary page. I have to go forward in the wizard again. And as I do so… guess what: the information that was missing during my first attempt is missing again. So there I go, filling-out bits of information a third time.

Now you can easily imagine how the software got to be so buggy: written with software tools not particularly designed to handle forms, developers had to write by hand things such as “when the user goes to page two from page three, restore such and such information from the session”. Somebody forgot to write the code that restores a particular piece of data, and lousy QA missed it. (Needless to say, there were not many usability tests on this wizard either.)

Now imagine the next version of this form if E*TRADE developers decide to use XForms and Orbeon Forms instead:

• The information gathered by the wizard is constantly kept in one or several XForms instances. The XForms engine takes care of keeping this around and not losing it.

• It is easy to check whether all the required information is entered correctly at any point in the wizard. This allows you to exit the workflow to the summary page from anywhere after you have amended information.

• You can easily provide multiple views of the same information, including the editable wizard view, and the read-only summary view: they both share the same underlying XML data representation.

• The wizard is automatically Ajax-enabled, with as-you-type validation, providing a much more pleasant user experience.

And of course, users are much happier because they enter their information once, not three times.

When a form field is loosing the focus, the browser is supposed to send two events: blur and change. The latter is only sent if the you have changed the value of the field. When you change the value of a first field and tab to a second field, all the browsers get the events right. But if you change the value of a first field, alt-tab to another application, alt-tab back to the browser, and tab to the second field, then different browsers implement different behaviors:

• IE 7 implements this “correctly”, i.e. as if the alt-tabs between the browser and the other application were not there.
• Firefox 2 on Windows sends a change event when you alt-tab away from Firefox, and then doesn’t send the change event when you tab to the second field. This is not what I would expect, but it isn’t terribly incorrect either.
• Firefox 2 on Mac never sends the change event. That is just plain wrong.

If you are using XForms to define your forms and running them through Orbeon Forms you won’t have to deal with any of this. Orbeon Forms will do the hard work for you. Lucky you! ;)

Over the last two weeks, we fixed a series of bugs that we considered blockers for Orbeon Forms 3.6:

• Error when using browser back button when using xforms:instance[@src = ‘input:*’]
  
• Label is not hidden when repeat iteration is added
  
• Inspector does not show initial XML (in some cases)
  
• Issue with adjacent itemsets
  
• Google Suggests example doesn’t work anymore
  
• XPL, XQuery, XSLT, XPath and XUpdate sandboxes don’t show results anymore
  
• Logout example: still shows the page as if logged in after logging out
  

We have one remaining JavaScript bug which we consider a blocker, and then we should be able to officially proceed to an RC build!