The adventures of XSLT 2.0 in Browserland

See also:


The purpose of my presentation at XML Amsterdam was to explore what might be done  with Saxon-CE in the domain of forms applications.

I won’t repeat here what I said during the presentation but focus on a few key points and give as much pointers as possible to explore if you want to go further.

For those of you who have missed my presentation or would like to hear it again, the next best thing is the screencast of my last rehearsal before the event:

The screencast of the last rehearsal of my presentation at XML Amsterdam

The presentation itself is powered by Saxon-CE and the best way if you want to try it by yourself is to download or clone it on our GitLab server.

An alternative is to browse it directly from this server. Note however that serving raw resources through GitLab is not very efficient and that the response times will be slow.

You may notice differences between the screencast, what I have presented, the demo showed by Geert Josten at the DemoJam and the latest version on GitLab.

A look at the network of changes should help you to understand what has happened! The version that I have presented is tagged as xmlamsterdam2013. The version used for the screencast was the previous one, before adding a page with metrics. I have wanted to play it safe and the features needed to implement Geert’s calculator have been developed separately in the grtjn branch which has been merged after the conference into the master branch…

Impedance mismatches

A key point to manipulate XML documents based on user interaction is the ability to store these documents in the browser between user actions and I have spent height slides, between slide 13 “Storing instance in an XML hostile environment” and slide 20 “Storing XML in the global (window) object” to explore different options.

It’s time to explain the issues we have to face to be able to store instances as JavaScript properties.

While Saxon-CE is fine within browsers and does a good job of acting as a good citizen speaking JavaScript almost as it if was its native language, there are still a number of glitches that reminds us that its isn’t really the case…

Among them, many castings which are implicit in XSLT don’t work when accessing JavaScript object properties or methods.

In XSLT, we use to write things such as:

    <xsl:template name="init">
      <xsl:variable name="element" as="element()">
      <div><xsl:value-of select="concat('Element value: ', $element)"/></div>

In the <xsl:value-of/> instruction, the second parameter to the concat() function is an element node which as been implicitly casted into a string.

As a naive user, I was thinking that I could do the same when calling JavaScript methods, for instance:

      <xsl:variable name="element" as="element()">
      <div><xsl:value-of select="js:alert($element)"/></div>

But the browser strongly disagreed:

SaxonCE.XSLT20Processor 20:25:07.774
SEVERE: XPathException in invokeTransform: Cannot convert class to Javascript object

This message comes to remind you that two type of incompatible objects live within Saxon-CE and that native Java classes cannot always be converted into JavaScript objects!

In this simple case we can explicitly cast the element to a simple type which will be compatible with JavaScript object system:

      <xsl:variable name="element" as="element()">
      <div><xsl:value-of select="js:alert(string($element))"/></div>

That’s more difficult if you really need to pass an XML node as a JavaScript parameter or property. For instance if I want to save this element as a property, this will raise the same kind of errors:

      <xsl:variable name="element" as="element()">
      <div><ixsl:set-property name="element" select="$element"/></div>

Here I need to convert the element node into a JavaScript object and I would even say that ideally this conversion should be implicit.

The bad news is that Saxon-CE does not provide a function to perform this conversion.

The other bad news is that if you thought you could use Saxon-CE JavaScript js:Saxon.serializeXML() and js:Saxon.parseXML() to perform that task by writing:

      <xsl:variable name="element" as="element()">
      <div><ixsl:set-property name="element" select="js:Saxon.parseXML(js:Saxon.serializeXML($element))"/></div>

you’re out of luck: js:Saxon.serializeXML() is a JavaScript wrapper to a native JavaScript function that won’t accept your XML node as an argument.

Michael Kay has implemented an ixsl:serialize-xml() function in branch ce1.1plus but unfortunately this function works fine with Saxon’s tiny tree nodes but silently fails when its parameter is a JavaScript DOM object. Furthermore, ixsl:serialize-xml() does not always preserve comments and PIs.

This conversion is one of the basis of most of the things I have shown during my presentation and I ended up writing a “d:serializeXML()” function, based on Evan Lenz’ xml-to-string.xsl and wrap this function in a js:Saxon.parseXML() call to provide the d:convert-to-jsdom() function that I needed:

      <xsl:variable name="element" as="element()">
      <div><ixsl:set-property name="element" select="d:convert-to-jsdom($element)"/></div>

This is probably a most inefficient way of performing the conversion. A slightly more efficient way could be to write an XSLT implementation of this function that would use JavaScript methods to create the DOM tree from within XSLT templates rather than serializing to parse again.

At the end of the day, I think that Saxon-CE should really provide a built-in method to perform the conversion and that this conversion should be implicit!

See also:

One transformation is not enough!

 When you love you don’t count the cost

Saxon-CE analyses the page to find the first script element with @type='application/xslt+xml' and @language='xslt2.0' in the HTML page to create an XSLT transformer on which it calls the updateHTMLDocument() method.

This a great idea and that’s perfect to provide an easy way to run a transformation acting as a set of event handlers on the page!

However, why should us be limited to a single transformation?

In my presentation I have found out that I had several disjoint set of features to implement on each slide and in such case I much prefer to write a transformation for each set of features.

The bad news is that Saxon CE won’t do that automatically for you.

The good news is that you can do that using the JavaScript API provided by Saxon-CE. And since you can call JavaScript methods in XSLT you can even do that in XSLT.

That’s what I did in my presentation: the first transformation which is automatically invoked by Saxon-CE is called boot-saxon.xsl. This transformation matches the following scripts linking to XSLT 2.0 transformations, creates an XSLT transformer and invokes the updateHTMLDocument() method…

If you look at this transformation you’ll see that in addition it can also invoke dynamic transformations (transformations which are created by transforming the web page into XSLT). This is a feature I use in my proof of concept of an XForms implementation.

Multiple transformations are a great feature and I think they would really deserve to become a standard feature in Saxon-CE.

Dynamic transformations are probably much more a niche thing and I am not sure they should be integrated to the product.

See also:

The beauty of XSLT as an event handler self-modifying document

updateHTMLDocument() is a misnomer

Beside bringing XSLT 2.0 to the browser, I think that the main innovation of Saxon-CE is the way it uses XSLT to define event handlers which update the current page based on interaction events.

In slide 26, “A dream come true”, I am using this method on an XSLT transformation run against an XML fragment and as you can see it seems to be working pretty well!

Why would we need to run this method on anything else than the HTML page?

This was one of the things I have explained in detail during my presentation: I think that the new paradigm introduced by Saxon-CE to use XSLT “transformations” as even handlers able to update document fragments is really powerful and has use cases well beyond “typical” Saxon-CE applications.

The purpose of this slide is to show how nice it would be to run a client side MVC application implemented as two separate transformations communicating through events:

  • a first one acting on the page, reacting to user interactions
  • a second one acting on an instance and controlling its updates.

I see no reason why this shouldn’t be extended to other use cases.

For instance, such a feature would open interesting perspective in XML databases where XSLT transformations could be used to define event handlers on XML nodes…

To come back to Saxon-CE, I think that both the name of the method and the documentation are misleading:

  • updateHTMLDocument() could be renamed updateDocument()
  • The description of the method’s $target parameter should read “The Document object to update” instead of “The HTML Document object to update”.

Events seem (too) strongly limited

This proposal to generalize the usage of the updateHTMLDocument() method assumes that events can be used on XML documents.

Unfortunately the usage of events in Saxon-CE seem to be strictly limited.

The documentation differentiates two different types of events:

  • User input events: Event handlers for user input are written in the form of template rules. The match pattern of the template rule must match the element that receives the event, and the mode name reflects the type of event, for example ixsl:onclick.
  • Client system events: Saxon-CE also handles events raised by objects such as window that live outside the DOM. Event handlers for such objects are written in the form of template rules. The match pattern is different from that for conventional templates because there is no node to match. Instead, the pattern must be an ixsl function (e.g. ixsl:window() ) that returns the object whose event is to be handled.

Unfortunately both types of events suffer important restrictions:

  1. User input events seem limited to a subset of standard HTML events as listed for instance by MDN. This limitation isn’t enforced by Saxon-CE itself but the fact is that most events are just ignored. This includes not only any custom event but events such as the input event. If you define a template with a mode matching an event which is not supported (such as ixsl:oninput), Saxon-CE will silently ignore this definition and won’t define any event handler.
  2. Furthermore, when using updateHTMLDocument() with an XML document as a target Saxon-CE ignores any event handler that you define through ixsl:onXXX modes.
  3. Client system events on the contrary do support any type of event. However I have never been able to use them with any JavaScript object except the global window object!
  4. Custom events sent to the window object using standard dispatch methods do not seem to work and it is safer to directly call the event handler than to dispatch events.

A consequence of the first limitation is that I haven’t found any way to implement the equivalent of XForms incremental mode. This should be trivial to implement using input events, I have checked that these events are fired synchronously when  an input is changed but since Saxon-CE doesn’t install any event handler for them there seems to be no way to catch them.

Another consequence is that as you may have noticed that in hello-world-2xslt.xsl we define an event handler on a change event for the output:

    <!-- The output needs to be updated -->
    <xsl:template match="div[@id='output']" mode="ixsl:onchange">
        <xsl:message>The output needs to be updated</xsl:message>
        <xsl:variable name="instance" select="ixsl:get(ixsl:window(), 'instance')"/>
        <xsl:result-document href="#output" method="ixsl:replace-content">
            <xsl:text>Hello </xsl:text>
            <xsl:value-of select="$instance/data/person-given-name"/>
            <xsl:text>. We hope you like Saxon CE!</xsl:text>

You may wonder how a change event can be sent to an HTML <div/>. In fact this event is sent as a custom event by the transformation implementing the model and it would have been much less confusing to use a custom name but because custom events on DOM nodes are ignored by Saxon-CE I had to hijack an existing HTML event and have chosen one that should never be fired to avoid and conflict with user interaction.

At that point you probably wonder how slide 26 can work if we can’t define event handlers for XML nodes (this is the second restriction listed above).

I had to find a workaround and ended up simulating proper event handling in functions defined in events.xsl.

The most controversial pieces in this library are probably:

  <xsl:function name="d:new-custom-event">
    <xsl:param name="name" as="xs:string"/>
    <xsl:variable name="js-statement" as="element()">
      <root statement="new CustomEvent ( '{$name}',{{ detail: {{}} }})"/>
    <xsl:sequence select="ixsl:eval(string($js-statement/@statement))"/>

  <xsl:function name="d:dispatch-event-to-instance">
    <xsl:param name="target"/>
    <xsl:param name="event"/>
    <ixsl:set-property object="$event" name="" select="$target"/>
    <ixsl:set-property name="dummy" select="ixsl:call(ixsl:window(), 'onEventWorkaround', $event)"/>

Instead of creating an event with the type defined as a parameter, d:new-custom-event() creates an event and stores the type as a property.

Instead of really dispatching an event to a target, d:dispatch-event-to-instance() stores the target as a property of the event and performs a direct call to the onEventWorkaround() event handler on the window object.

This event handler is defined as:

  <xsl:template match="ixsl:window()" mode="ixsl:onEventWorkaround">
    <xsl:message>Got en event</xsl:message>
    <xsl:variable name="event" select="ixsl:event()"/>
      <xsl:when test="ixsl:get($event, 'type') = 'ModelUpdate'">
        <!--<xsl:apply-templates select="ixsl:get(event, '')" mode="d:onModelUpdate">-->
          select="ixsl:get(ixsl:window(), 'instance')//*[d:path(.) = d:path(ixsl:get($event, ''))]"
          <xsl:with-param name="event" select="$event"/>

This single event handler serves as a channel for events sent to the XML instance. To simulate the standard behavior of event handling in Saxon-CE, it applies the templates on the target passed as an event property. The biggest downside of this workaround is probably that this isn’t really extensible and that new <xsl:when/> need to be added for each supported event!

In find these four issues really problematic and think they deserve to be further analyzed and fixed!

See also:

Side effect

This point is briefly mentioned in slide 23, “Side effect”.

XSLT has been designed to be free of side effect and to develop this kind of user interface we rely very heavily on side effect.

Among other side effects, we often set properties on JavaScript objects.

In hello-world-mvc.xsl for instance (used in slide 21, “Complete example (when all we have is a hammer…)”) we write:

        <!-- Store the new value -->
        <ixsl:set-property object="ixsl:window()" name="instance"
        <!-- Rely on the instance to write the output -->
        <xsl:result-document href="#output" method="ixsl:replace-content">
            <xsl:text>Hello </xsl:text>
            <xsl:value-of select="$instance/data/person-given-name"/>
            <xsl:text>. We hope you like Saxon CE!</xsl:text>

XSLT being side effect free, there is nothing in the recommendation that insures that the <xsl:result-document/> will be executed after the <ixsl:property/>.

It happens to work but we need to realize that we are here on the edges of what should be done with XSLT and, of course, hope that Saxon-CE won’t break our assumption that in most of the cases XSLT instructions will be executed in document order!


XML Prague 2011 : XML à l’attaque du web

Salle de conférence pendant la pause café

Après une période un peu folle entre 2000 et 2008 pendant laquelle j’ai participé à un nombre impressionnant de conférences, je m’étais mis un peu en retrait et n’avais plus participé à aucune conférence depuis XTech 2008.

XML Prague 2011 était donc pour moi l’occasion de rencontrer à nouveau la communauté des experts XML internationaux et j’étais curieux de voir comment elle avait évolué pendant ces trois dernières années.

MURATA Makoto (EPUB3: Global Language and Comic)A côté des aspects plus techniques, je n’oublierai pas l’image de Murata Makoto exprimant sobrement sa peine pour les victimes du tremblement de terre au Japon.

La tagline de XML Prague 2011 était “XML devait être l’espéranto du Web. Pourquoi n’est-ce pas le cas?” (“XML as new lingua franca for the Web. Why did it never happen?”).

Michael Sperberg-McQueen (Closing keynote)Le contenu de la conférence est resté proche de cette ligne mais il a été résumé de manière plus exacte par Michael Sperberg-McQueen lors de sa clôture : “Mettons du XML dans le navigateur, qu’ils le veuillent ou non!”

Norman Walsh (HTML+XML: The W3C HTML/XML Task Force)Le ton a été donné par Norman Walsh dès la toute première présentation: la convergence entre HTML et XML n’aura pas lieu.

XML a tenté d’être un format neutre convenant aussi bien aux documents qu’aux données sur le web. On peut dire aujourd’hui que cet objectif n’a pas été atteint et que les formats les plus populaires sur le web sont HTML pour les documents et JSON pour les données.

Cela ne semble pas préoccuper plus que mesure le public de XML Prague composé d’aficionados des langages à balises : si la “masse des développeurs web” n’est pas intéressée par XML c’est son problème. Les bénéfices liés à XML sont bien connus et cela signifie simplement que la communauté XML devra développer les outils nécessaires pour utiliser XML dasn le navigateur aussi bien que sur le serveur.

Sur ce thème, beaucoup de présentations couvraient le support de XML dans le navigateur ainsi que les passerelles entre JSON et XML :

  • Validation XML Schema côté client par Henry S. Thompson and Aleksejs Goremikins
  • JSON pour XForms par Alain Couthures
  • XSLT dans le navigateur par Michael Kay
  • Traitement de XML efficace dans les navigateurs par Alex Milowski
  • XQuery dans le navigateur par Peter Fischer

Les outils côté serveurs ont fait l’objet de moins de sessions, peut être parce que le sujet est plus ancien :

  • Une façade JSON pour le serveur MarkLogic par Jason Hunter
  • CXAN: étude de cas pour Servlex, un framework XML pour le web par Florent Georges
  • Akara – “Spicy Bean Fritters” et services de données XML par Uche Ogbuji

Bien entendu, les standards étaient aussi au programme :

  • HTML+XML: la task force W3C HTML/XML (déjà mentionnée) par Norman Walsh
  • Standards update: XSLT 3.0 par Michael Kay
  • Standards update: XML, XQuery, XML Processing Profiles, DSDL par Liam Quin, Henry S. Thompson, Jirka Kosek

Ainsi que les applications de XML :

  • Configuration d’équipements réseau avec NETCONF et YANG par Ladislav Lhotka
  • Développements XML – XML Projects par George Bina
  • EPUB3: le langage et les bandes dessinées par Murata Makoto
  • EPUB: Chapitres et versets par Tony Graham
  • DITA NG – une implémentation Relax NG de DITA par George Bina

Sans oublier quelques présentations techniques sur les implémentations elles mêmes :

  • Traduction de SPARQL et SQL en XQuery par Martin Kaufmann
  • Réécritures déclaratives de XQuery pour le profit et le plaisir par John Snelson

Et la séance de clôture par le roi de cet excercice, Michael Sperberg-McQueen.

Ma présentation, “injection XQuery”, était assez atypique dans cet ensemble et il a fallu tout le talent de Michael Sperberg-McQueen pour lui trouver un point commun en faisant remarquer que pour avoir une chance de mettre XML sur le web il faudrait se préoccuper un peu plus de sécurité.

J’avais été impressionné lors des conférences XTech par l’évolution des techniques de présentation, la plupart des intervenants rejetant les traditionnelles présentation powerpoint et leurs “transparents” surchargés pour des alternatives plus légères et beaucoup plus imagées.

John Snelson (Declarative XQuery Rewrites for Profit or Pleasure)Je pensais ce mouvement inéluctable et ai été bien surpris de voir qu’il n’avait guère atteint les intervenants de XML Prague 2011 qui (à l’exception très notable de John Snelson) continuaient à utiliser powerpoint de manière très traditionnelle.

J’avais conçu ma présentation en suivant ce que je croyais être la technique de présentation devenue classique. Utilisant Slidy, j’avais pas moins de 35 pages très concises à présenter en 25 minutes. Chaque page avait une photo différente en arrière plan et ne comprenait que quelques mots.

Les commentaires ont été plutôt positifs bien que certaines photos d’injections aient choqué quelques participants.

Ma présentation étant du HTML standard, j’avais jugé plus sur d’utiliser l’ordinateur mis à disposition par les organisateurs. C’était sans compter sur les 74 Moctets d’images à charger pour les fonds de pages qui ont mis à mal cet ordinateur un peu poussif et les pages étaient un peu lentes à l’affichage (note personnelle : la prochaine fois, utilise ton ordinateur)!

The twitter wall (and Norman Walsh)Le “mur twitter” projeté au moyen d’un second vidéo projecteur a eu également beaucoup de succès.

Ce mur a été bien pratique pour communiquer pendant les sessions et il remplace avantageusement les canaux IRC que nous utilisions auparavant.

Twitter ne permet malheureusement pas de rechercher dans ses archives et, alors que j’écris ces mots, je ne peux déjà plus accéder aux tweets du premier jour de la conférence!

Avec un peu de recul, si j’essaye d’analyser ce qui s’est dit à XML Prague 2011, j’ai des sentiments mitigés à propos de ce fossé qui se creuse entre communautés Web et XML.

Le rêve que XML puisse être accepté par l’ensemble de la communauté des développeurs web était une vision très forte et nous ne devons pas oublier que XML a été conçu pour mettre “SGML sur le web“.

Ceci dit, il faut bien reconnaître que les développeurs web ont toujours été réticents devant la complexité additionnelle (réelle ou perçue) de XHTML. Ce fossé a toujours existé et après que XML ait manqué le virage du Web 2.0 il était trop tard pour espérer le combler.

XML sur le web restera donc une niche et continuera à être utilisé par une minorité, mais la créativité et le dynamisme de la communauté qui s’est manifesté à Prague est impressionnant et encourageant : il y a encore place pour beaucoup d’innovations et XML est, plus que jamais, une technologie de choix pour développer des applications web.


XML Prague 2011: XML against the web

Coffee break

After a frenzy period between 2000 and 2008 where I have spoken at an impressive number of conferences, I temporally retired and hadn’t been at a conference since XTech 2008.

For me, XML Prague 2011 was the first opportunity to meet again face to face with the community of XML core techies and I was curious to find out what the evolution had been during the past three years.

MURATA Makoto (EPUB3: Global Language and Comic)Aside from all the technical food for thought, an image of the conference that I won’t forget is Murata Makoto expressing his grief for the victims of the earthquake in Japan with simple and sober terms.

The tag line of XML Prague 2011 was “XML as new lingua franca for the Web. Why did it never happen?”.

Michael Sperberg-McQueen (Closing keynote)The actual content of the conference has been close to this tag line but was better summarized by Michael Sperberg-McQueen during his closing keynotes: “Let’s put XML in the browser, whether they want it there or not!”

Norman Walsh (HTML+XML: The W3C HTML/XML Task Force)The tone was given by Norman Walsh during the very first session: the convergence between HTML and XML will not happen.

XML has been trying hard to be an application neutral format for the web that could be used both for documents and data. It is fair to say that it has failed to reach this goal and that the preferred formats on the web are HTML for documents and JSON for data.

That doesn’t seem to bother that much the XML Prague attendees who are markup language addicts anyway: if the “mass of web developers” do not care about XML that’s their problem. The benefits of using XML is well known and that just means that we have to develop the XML tools we need on the server as well as on the browser.

Following this line, many sessions were about developing XML support on the browser and bridging the gaps between XML and HTML/JSON:

  • Client-side XML Schema validation by Henry S. Thompson and Aleksejs Goremikins
  • JSON for XForms by Alain Couthures
  • XSLT on the browser by Michael Kay
  • Efficient XML Processing in Browsers by Alex Milowski
  • XQuery in the Browser reloaded by Peter Fischer

By contrast, server side tools have been less represented, maybe because the domain had been better covered in the past:

  • A JSON Facade on MarkLogic Server by Jason Hunter
  • CXAN: a case-study for Servlex, an XML web framework by Florent Georges
  • Akara – Spicy Bean Fritters and XML Data Services by Uche Ogbuji

Of course, standard updates were also on the program:

  • HTML+XML: The W3C HTML/XML Task Force (already mentioned) by Norman Walsh
  • Standards update: XSLT 3.0 by Michael Kay
  • Standards update: XML, XQuery, XML Processing Profiles, DSDL by Liam Quin, Henry S. Thompson, Jirka Kosek

We also had talks about XML applications:

  • Configuring Network Devices with NETCONF and YANG by Ladislav Lhotka
  • Advanced XML development – XML Projects by George Bina
  • EPUB3: Global Language and Comic by Murata Makoto
  • EPUB: Chapter and Verse by Tony Graham
  • DITA NG – A Relax NG implementation of DITA by George Bina

Without forgetting a couple of implementation considerations:

  • Translating SPARQL and SQL to XQuery by Martin Kaufmann
  • Declarative XQuery Rewrites for Profit or Pleasure by John Snelson

And the traditional and always impressive closing keynote by Michael Sperberg-McQueen.

My own presentation, “XQuery injection”, was quite atypical and it took all the talent of Michael Sperberg-McQueen to kindly relate it to “XML on the web” by noticing that security would have to be taken more seriously to make it happen.

One of the things that had impressed me during XTech conferences was the shift in presentation styles, most speakers moving away from heavy bullet points stuffed traditional powerpoint presentations to lighter and better illustrated shows.

John Snelson (Declarative XQuery Rewrites for Profit or Pleasure)I had expected the move to continue and have been surprised to see that the movement doesn’t seem to have caught XML Prague presenters whom continued to do with traditional bullet points with only a couple of exceptions (John Snelson being a notable exception).

I had worked my presentation to use what I thought would be a common style. Using Slidy, I had created no less than 35 short pages to present in 25 minutes. Each page had a different high resolution picture as a background and contained only a few words.

The comments have been generally good even though some pictures chosen to represent injections seem to have hurt the feelings of some attendees.

Since my presentation is just standard HTML, I had been brave enough to use the shared computer. Unfortunately, the presentation loads 74 Megs of background pictures and that was a little bit high for the shared computer that took several seconds to change pages (note to self: next time, use your own laptop)!

The twitter wall (and Norman Walsh)Another interesting feature of this conference was the “twitter wall” that was projected in the room using a second video projector.

This wall has proven to be very handy to communicate during the sessions and it can be seen like a more modern incarnation of the IRC channels used in earlier conferences.

Unfortunately, twitter doesn’t allow to search in archives and while I am writing these words, I can no longer go back in the past and read the tweets of the first day of the conference.

Looking backward at the conference, I have mixed feelings about this gap that now seems to be widely accepted on both sides between the XML and the web developers communities.

The dream that XML could be accepted by the web community at large was a nice vision and we should not forget that XML has been designed to be “SGML on the web“.

Web developers have always been reluctant to accept the perceived additional complexity of XHTML and the gap has been there from the beginning and after XML missed the train of Web 2.0 it was too late to close it.

XML on the web will stay a niche and will be used by a minority but the creativity and dynamism of the community shown at Prague is inspiring and encouraging: there is still room for a lot of innovation and XML is more than ever the technology of choice to power web applications.

All the pictures