Bill de hÓra has a blog post entitled Format Debt: what you can't say where he writes

The closest thing to a deployable web technology that might improve describing these kind of data mashups without parsing at any cost or patching is RDF. Once RDF is parsed it becomes a well defined graph structure - albeit not a structure most web programmers will be used to, it is however the same structure regardless of the source syntax or the code and the graph structure is closed under all allowed operations.

If we take the example of MediaRSS, which is not consistenly used or placed in syndication and API formats, that class of problem more or less evaporates via RDF. Likewise if we take the current Zoo of contact formats and our seeming inability to commit to one, RDF/OWL can enable a declarative mapping between them. Mapping can reduce the number of man years it takes to define a "standard" format by not having to bother unifying "standards" or getting away with a few thousand less test cases. 

I've always found this particular argument by RDF proponents to be suspect. When I complained about the the lack of standards for representing rich media in Atom feeds, the thrust of the complaint is that you can't just plugin a feed from Picassa into a service that understands how to process feeds from Zooomr without making changes to the service or the input feed.

RDF proponents  often to argue that if we all used RDF based formats then instead of having to change your code to support every new photo site's Atom feed with custom extensions, you could instead create a mapping from the format you don't understand to the one you do using something like the OWL Web Ontology Language.  The problem with this argument is that there is a declarative approach to mapping between XML data formats without having to boil the ocean by convincing everyone to switch to RD; XSL Transformations (XSLT).

The key problem is that in both cases (i.e. mapping with OWL vs. mapping with XSLT) there is still the problem that Picassa feeds won't work with an app that understand's Zoomr's feeds until some developer writes code. Thus we're really debating on whether it is better cheaper to have the developer write declarative mappings like OWL or XSLT instead of writing new parsing code in their language of choice.

In my experience I've seen that creating a software system where you can drop in an XSLT, OWL or other declarative mapping document to deal with new data formats is cheaper and likely to be less error prone than having to alter parsing code written in C#, Python, Ruby or whatever. However we don't need RDF or other Semantic Web technologies to build such solution today. XSLT works just fine as a tool for solving exactly that problem.

Now Playing: Lady GaGa & Colby O'Donis - Just Dance

 

Via Mark Pilgrim I stumbled on an article by Scott Loganbill entitled Google’s Open Source Protocol Buffers Offer Scalability, Speed which contains the following excerpt

The best way to explore Protocol Buffers is to compare it to its alternative. What do Protocol Buffers have that XML doesn’t? As the Google Protocol Buffer blog post mentions, XML isn’t scalable:

"As nice as XML is, it isn’t going to be efficient enough for [Google’s] scale. When all of your machines and network links are running at capacity, XML is an extremely expensive proposition. Not to mention, writing code to work with the DOM tree can sometimes become unwieldy."

We’ve never had to deal with XML in a scale where programming for it would become unwieldy, but we’ll take Google’s word for it.

Perhaps the biggest value-add of Protocol Buffers to the development community is as a method of dealing with scalability before it is necessary. The biggest developing drain of any start-up is success. How do you prepare for the onslaught of visitors companies such as Google or Twitter have experienced? Scaling for numbers takes critical development time, usually at a juncture where you should be introducing much-needed features to stay ahead of competition rather than paralyzing feature development to keep your servers running.

Over time, Google has tackled the problem of communication between platforms with Protocol Buffers and data storage with Big Table. Protocol Buffers is the first open release of the technology making Google tick, although you can utilize Big Table with App Engine.

It is unfortunate that it is now commonplace for people to throw around terms like "scaling" and "scalability" in technical discussions without actually explaining what they mean. Having a Web application that scales means that your application can handle becoming popular or being more popular than it is today in a cost effective manner. Depending on your class of Web application, there are different technologies that have been proven to help Web sites handle significantly higher traffic than they normally would. However there is no silver bullet.

The fact that Google uses MapReduce and BigTable to solve problems in a particular problem space does not mean those technologies work well in others. MapReduce isn't terribly useful if you are building an instant messaging service. Similarly, if you are building an email service you want an infrastructure based on message queuing not BigTable. A binary wire format like Protocol Buffers is a smart idea if your applications bottleneck is network bandwidth or CPU used when serializing/deserializing XML.  As part of building their search engine Google has to cache a significant chunk of the World Wide Web and then perform data intensive operations on that data. In Google's scenarios, the network bandwidth utilized when transferring the massive amounts of data they process can actually be the bottleneck. Hence inventing a technology like Protocol Buffers became a necessity. However, that isn't Twitter's problem so a technology like Protocol Buffers isn't going to "help them scale". Twitter's problems have been clearly spelled out by the development team and nowhere is network bandwidth called out as a culprit.

Almost every technology that has been loudly proclaimed as unscalable by some pundit on the Web is being used by a massively popular service in some context. Relational databases don't scale? Well, eBay seems to be doing OK. PHP doesn't scale? I believe it scales well enough for Facebook. Microsoft technologies aren't scalable? MySpace begs to differ. And so on…

If someone tells you "technology X doesn't scale" without qualifying that statement, it often means the person either doesn't know what he is talking about or is trying to sell you something. Technologies don't scale, services do. Thinking you can just sprinkle a technology on your service and make it scale is the kind of thinking that led Blaine Cook (former architect at Twitter) to publish a presentation on Scaling Twitter which claimed their scaling problems where solved with their adoption of memcached. That was in 2007. In 2008, let's just say the Fail Whale begs to differ. 

If a service doesn't scale it is more likely due to bad design than to technology choice. Remember that.

Now Playing: Zapp & Roger - Computer Love

 

Jeff Atwood recently published two anti-XML rants in his blog entitled XML: The Angle Bracket Tax and Revisiting the XML Angle Bracket Tax. The source of his beef with XML and his recommendations to developers are excerpted below

Everywhere I look, programmers and programming tools seem to have standardized on XML. Configuration files, build scripts, local data storage, code comments, project files, you name it -- if it's stored in a text file and needs to be retrieved and parsed, it's probably XML. I realize that we have to use something to represent reasonably human readable data stored in a text file, but XML sometimes feels an awful lot like using an enormous sledgehammer to drive common household nails.

I'm deeply ambivalent about XML. I'm reminded of this Winston Churchill quote:

It has been said that democracy is the worst form of government except all the others that have been tried.

XML is like democracy. Sometimes it even works. On the other hand, it also means we end up with stuff like this:

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" 
  SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
  <SOAP-ENV:Body>
    <m:GetLastTradePrice xmlns:m="Some-URI">
      <symbol>DIS</symbol>
    </m:GetLastTradePrice>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

…
You could do worse than XML. It's a reasonable choice, and if you're going to use XML, then at least learn to use it correctly. But consider:

1. Should XML be the default choice?
2. Is XML the simplest possible thing that can work for your intended use?
3. Do you know what the XML alternatives are?
4. Wouldn't it be nice to have easily readable, understandable data and configuration files, without all those sharp, pointy angle brackets jabbing you directly in your ever-lovin' eyeballs?

I don't necessarily think XML sucks, but the mindless, blanket application of XML as a dessert topping and a floor wax certainly does. Like all tools, it's a question of how you use it. Please think twice before subjecting yourself, your fellow programmers, and your users to the XML angle bracket tax. <CleverEndQuote>Again.</CleverEndQuote>

The question of if and when to use XML is one I am intimately familiar with given that I spent the first 2.5 years of my professional career at Microsoft working on the XML team as the “face of XML” on MSDN.

My problem with Jeff’s articles is that they take a very narrow view of how to evaluate a technology. No one should argue that XML is the simplest or most efficient technology to satisfy the uses it has been put to today. It isn’t. The value of XML isn’t in its simplicity or its efficiency. It is in the fact that there is a massive ecosystem of knowledge and tools around working with XML.

If I decide to use XML for my data format, I can be sure that my data will be consumable using a variety off-the-shelf tools on practically every platform in use today. In addition, there are a variety of tools for authoring XML, transforming it to HTML or text, parsing it, converting it to objects, mapping it to database schemas, validating it against a schema, and so on. Want to convert my XML config file into a pretty HTML page? I can use XSLT or CSS. Want to validate my XML against a schema? I have my choice of Schematron, Relax NG and XSD. Want to find stuff in my XML document? XPath and XQuery to the rescue. And so on.

No other data format hits a similar sweet spot when it comes to ease of use, popularity and breadth of tool ecosystem.

So the question you really want to ask yourself before taking on the “Angle Bracket Tax” as Jeff Atwood puts it, is whether the benefits of avoiding XML outweigh the costs of giving up the tool ecosystem of XML and the familiarity that practically every developer out there has with the technology? In some cases this might be true such as when deciding whether to go with JSON over XML in AJAX applications (I’ve given two reasons in the past why JSON is a better choice).  On the other hand, I can’t imagine a good reason to want to roll your own data format for office documents or application configuration files as opposed to using XML.

FURTHER READING

• The XML Litmus Test - Dare Obasanjo provides some simple guidelines for determining when XML is the appropriate technology to use in a software application or architecture design. (6 printed pages)
• Understanding XML - Learn how the Extensible Markup Language (XML) facilitates universal data access. XML is a plain-text, Unicode-based meta-language: a language for defining markup languages. It is not tied to any programming language, operating system, or software vendor. XML provides access to a plethora of technologies for manipulating, structuring, transforming and querying data. (14 printed pages)

Now Playing: Metallica - The God That Failed

 

Disclaimer: This may sound like a rant but it isn't meant to be. In the wise words of Raymond Chen this is meant to highlight problems that are harming the productivity of developers and knowledge workers in today's world. No companies or programs will be named because the intent is not to mock or ridicule. 

This morning I had to rush into work early instead of going to the gym because of two limitations in the software around us.

Problem #1: Collaborative Document Editing

So a bunch are working on a document that is due today. Yesterday I wanted to edit the document but found out I could not because the software claimed someone else was currently editing the document. So I opened it in read-only mode, copied out some data, edited it and then sent my changes in an email to the person who was in charge of the document. As if that wasn’t bad enough…

This morning, as I'm driving to the gym for my morning work out, I glance at my phone to see that I've received mail from several co-workers because it I've "locked" the document and no one can make their changes. When I get to work, I find out that I didn’t close the document within the application and this was the reason none of my co-workers could edit it. Wow.

The notion that only one person at a time can edit a document or that if one is viewing a document, it cannot be edited seems archaic in today’s globally networked world. Why is software taking so long to catch up?

Problem #2: Loosely Coupled XML Web Services

While I was driving to the office I noticed another email from one of the services that integrates with ours via a SOAP-based XML Web Service. As part of the design to handle a news scenario we added a new type that was going to be returned by one of our methods (e.g. imagine that there was a GetFruit() method which used to return apples and oranges which now returns apples, oranges and bananas) . This change was crashing the applications that were invoking our service because they weren’t expecting us to return bananas.

However, the insidious thing is that the failure wasn’t because their application was improperly coded to fail if it saw a fruit it didn’t know, it was because the platform they built on was statically typed. Specifically, the Web Services platform automatically converted the XML to objects by looking at our WSDL file (i.e. the interface definition language which stated up front which types are returned by our service) . So this meant that any time new types were added to our service, our WSDL file would be updated and any application invoking our service which was built on a Web services platform that performed such XML<->object mapping and was statically typed would need to be recompiled. Yes, recompiled.

Now, consider how many potentially different applications that could be accessing our service. What are our choices? Come up with GetFruitEx() or GetFruit2() methods so we don’t break old clients? Go over our web server logs and try to track down every application that has accessed our service? Never introduce new types? 

It’s sad that as an industry we built a technology on an eXtensible Markup Language (XML) and our first instinct was to make it as inflexible as technology that is two decades old which was never meant to scale to a global network like the World Wide Web. 

Software should solve problems, not create new ones which require more technology to fix.

Now playing: Young Jeezy - Bang (feat. T.I. & Lil Scrappy)

 

Andy Conrad, who I used to work with back on the XML team, has two blog posts about Project Astoria and Project Jasper from Microsoft's Data Programmability team. Both projects are listed as data access incubation projects on MSDN. Below are the descriptions of the projects

Project Codename “Astoria”
The goal of Microsoft Codename Astoria is to enable applications to expose data as a data service that can be consumed by web clients within a corporate network and across the internet. The data service is reachable over regular HTTP requests, and standard HTTP verbs such as GET, POST, PUT and DELETE are used to perform operations against the service. The payload format for the service is controllable by the application, but all options are simple, open formats such as plan XML and JSON. Web-friendly technologies make Astoria an ideal data back-end for AJAX-style applications, and other applications that need to operate against data that is across the web.

To learn more about Project Astoria or download the CTP, visit the Project Astoria website at http://astoria.mslivelabs.com.

Project Codename “Jasper”
Project Jasper is geared towards iterative and agile development. You can start interacting with the data in your database without having to create mapping files or define classes. You can build user interfaces by naming controls according to your model without worrying about binding code. Project Jasper is also extensible, allowing you to provide your own business logic and class model. Since Project Jasper is built on top of the ADO.NET Entity Framework, it supports rich queries and complex mapping.

To learn more about Project Jasper visit the ADO.NET Blog at http://blogs.msdn.com/adonet

I was called in a few weeks ago by an architect on the Data Programmability team to give some advice about Project Astoria. The project is basically a way to create RESTful endpoints on top of a SQL Server database then retrieve the relational data as plain XML, JSON or a subset of RDF+XML using HTTP requests. The reason I was called in was to give some of my thoughts on exposing relational data as RSS/Atom feeds. My feedback was that attempting to map arbitrary relational data to RSS/Atom feeds seemed unnatural and was bordering on abuse of an XML syndication format. Although this feature was not included in the Project Astoria CTP, it seems that mapping relational data to RSS/Atom feeds is still something the team thinks is interesting based on the Project Astoria FAQ. You can find out more in the Project Astoria overview documentation.  

REST is totally sweeping Microsoft.

 

Miguel de Icaza of Gnumeric, GNOME and Ximian fame has weighed in with his thoughts on the FUD war that is ODF vs. OOXML. In his blog post entitled The EU Prosecutors are Wrong Miguel writes

Open standards and the need for public access to information was a strong message. This became a key component of promoting open office, and open source software. This posed two problems:

First, those promoting open standards did not stress the importance of having a fully open source implementation of an office suite. Second, it assumed that Microsoft would stand still and would not react to this new change in the market.

And that is where the strategy to promote the open source office suite is running into problems. Microsoft did not stand still. It reacted to this new requirement by creating a file format of its own, the OOXML.
...

The Size of OOXML

A common objection to OOXML is that the specification is "too big", that 6,000 pages is a bit too much for a specification and that this would prevent third parties from implementing support for the standard. Considering that for years we, the open source community, have been trying to extract as much information about protocols and file formats from Microsoft, this is actually a good thing.

For example, many years ago, when I was working on Gnumeric, one of the issues that we ran into was that the actual descriptions for functions and formulas in Excel was not entirely accurate from the public books you could buy.

OOXML devotes 324 pages of the standard to document the formulas and functions. The original submission to the ECMA TC45 working group did not have any of this information. Jody Goldberg and Michael Meeks that represented Novell at the TC45 requested the information and it eventually made it into the standards. I consider this a win, and I consider those 324 extra pages a win for everyone (almost half the size of the ODF standard).

Depending on how you count, ODF has 4 to 10 pages devoted to it. There is no way you could build a spreadsheet software based on this specification.
...
I have obviously not read the entire specification, and am biased towards what I have seen in the spreadsheet angle. But considering that it is impossible to implement a spreadsheet program based on ODF, am convinced that the analysis done by those opposing OOXML is incredibly shallow, the burden is on them to prove that ODF is "enough" to implement from scratch alternative applications.
...
The real challenge today that open source faces in the office space is that some administrations might choose to move from the binary office formats to the OOXML formats and that "open standards" will not play a role in promoting OpenOffice.org nor open source.

What is worse is that even if people manage to stop OOXML from becoming an ISO standard it will be an ephemeral victory.

We need to recognize that this is the problem. Instead of trying to bury OOXML, which amounts to covering the sun with your finger.

I think there is an interesting bit of insight in Miguel's post which I highlighted in red font. IBM and the rest of the ODF opponents lobbied governments against Microsoft's products by arguing that its file formats where not open. However they did not expect that Microsoft would turn around and make those very file formats open and instead compete on innovation in the user experience.

Now ODF proponents like Rob Weir who've been trumpeting the value of open standards now find themselves in the absurd position of arguing that is a bad thing for Microsoft to open up its file formats and provide exhaustive documentation for them. Instead they demand that Microsoft  should either  abandon backwards compatibility with the billions of documents produced by Microsoft Office in the past decade or that it should embrace and extend ODF to meet its needs. Neither of which sounds like a good thing for customers. 

I guess it's like Tim Bray said, life gets complicated when there are billion$ of dollars on the line. I'm curious to see how Rob Weir responds to Miguel's post. Ideally, we'll eventually move away from these absurd discussions about whether it is a bad thing for Microsoft to open up its file formats and hand them over to an international standards body to talking about how we office productivity software can improve the lives of workers by innovating on features especially with regards to collaboration in the workplace.  After all everyone knows that single user, office productivity software is dead. Right?

 

In response to my recent post entitled ODF vs. OOXML on Wikipedia one of my readers pointed out

Well, many of Weir's points are not about OOXML being a "second", and therefore unnecessary, standard. Many of them, I think, are about how crappy the standard actually is.

Since I don't regularly read Rob Weir's blog this was interesting to me. I wondered why someone who identifies himself as working for IBM on various ODF technical topics would be spending a lot of his time attacking a related standard as opposed to talking about the technology he worked. I assumed my reader was mistaken and decided to subscribe to his feed and see how many of his recent posts were about OOXML. Below is a screenshot of what his feed looks like when I subscribed to it in RSS Bandit a few minutes ago

Of his 24 most recent posts, 16 of them are explicitly about OOXML while 7 of them are about ODF.

Interesting. I wonder why a senior technical guy at IBM is spending more time attacking a technology whose proponents have claimed is not competitive with it instead of talking about the technology he works on? Reading the blogs of Microsoft folks like Raymond Chen, Jensen Harris or Brian Jones you don't see them dedicating two thirds of their blog postings to bash rival products or technologies.

From my perspective as an outsider in this debate it seems to me that OOXML is an overspecified description of an open XML document format that is backwards compatible with the billions of documents produced in Microsoft Office formats over the past decade. On the other hand, ODF is an open XML document format that aims to be a generic format for storing business documents that isn't tied to any one product which still needs some work to do in beefing up the specification in certain areas if interoperability is key.

In an ideal world both of these efforts would be trying to learn from each other. However it seems that for whatever reasons IBM has decided that it would rather that Microsoft failed at its attempt to open up the XML formats behind the most popular office productivity software in the world. How this is a good thing for Microsoft's customers or IBM's is lost on me.

Having a family member who is in politics, I've learned that whenever you see what seems like a religious fundamentalism there usually is a quest for money and/or power behind it. Reading articles such as Reader Beware as ODF News Coverage Increases it seems clear that IBM has a lot of money riding on being first to market with ODF-enabled products while simultaneously encouraging governments to only mandate ODF. The fly in the ointment is that the requirement of most governments is that the document format is open, not that it is ODF. Which explains IBM's unfortunate FUD campaign. 

Usually, I wouldn't care about something like this since this is Big Business and Politics 101, but there was something that Rick Jellife wrote in his post An interesting offer: get paid to contribute to Wikipedia which is excerpted below

So I think there are distinguishing features for OOXML, and one of the more political issues is do we want to encourage and reward MS for taking the step of opening up their file formats, at last?

The last thing I'd personally hate is for this experience to have soured Microsoft from opening up its technologies so I thought I'd throw my hat in the ring at least this once.

PS: It's pretty impressive that a Google search for "ooxml" pulls up a bunch of negative blog posts and the wikipedia article as the first couple of hits. It seems the folks on the Microsoft Office team need to do some SEO to fix that pronto.

 

This morning I stumbled upon an interestingly titled post by Rick Jellife which piqued my interest entitled An interesting offer: get paid to contribute to Wikipedia where he writes

I’m not a Microsoft hater at all, its just that I’ve swum in a different stream. Readers of this blog will know that I have differing views on standards to some Microsoft people at least.
...
So I was a little surprised to receive email a couple of days ago from Microsoft saying they wanted to contract someone independent but friendly (me) for a couple of days to provide more balance on Wikipedia concerning ODF/OOXML. I am hardly the poster boy of Microsoft partisanship! Apparently they are frustrated at the amount of spin from some ODF stakeholders on Wikipedia and blogs.

I think I’ll accept it: FUD enrages me and MS certainly are not hiring me to add any pro-MS FUD, just to correct any errors I see.
...
Just scanning quickly the Wikipedia entry I see one example straight away: The OOXML specification requires conforming implementations to accept and understand various legacy office applications . But the conformance section to the ISO standard (which is only about page four) specifies conformance in terms of being able to accept the grammar, use the standard semantics for the bits you implement, and document where you do something different. The bits you don’t implement are no-one’s business. So that entry is simply wrong. The same myth comes up in the form “You have to implement all 6000 pages or Microsoft will sue you.” Are we idiots?

Now I certainly think there are some good issues to consider with ODF versus OOXML, and it is good that they come out an get discussed. For example, the proposition that “ODF and OOXML are both office document formats: why should there be two standards?” is one that should be discussed. As I have mentioned before on this blog, I think OOXML has attributes that distinguish it: ODF has simply not been designed with the goal of being able to represent all the information possible in an MS Office document; this makes it poorer for archiving but paradoxically may make it better for level-playing-field, inter-organization document interchange. But the archiving community deserves support just as much as the document distribution community. And XHTML is better than both for simple documents. And PDF still has a role. And specific markup trumps all of them, where it is possible. So I think there are distinguishing features for OOXML, and one of the more political issues is do we want to encourage and reward MS for taking the step of opening up their file formats, at last?

I'm glad to hear that Rick Jellife is considering taking this contract. Protecting your brand on Wikipedia, especially against well-funded or organized detractors is unfortunately a full time job and one that really should be performed by an impartial party not a biased one. It's great to see that Microsoft isn't only savvy enough to realize that keeping an eye on Wikipedia entries about itself is important but also is seeking objective 3rd parties to do the policing.

It looks to me that online discussion around XML formats for business documents has significantly detoriorated. When I read posts like Rob Weir's A Foolish Inconsistency and The Vast Blue-Wing Conspiracy or Brian Jones's Passing the OpenXML standard over to ISO it seems clear that rational technical discussion is out the windows and the parties involved are in full mud slinging mode. It reminds me of watching TV during U.S. election years. I'm probably a biased party but I think the "why should we have two XML formats for business documents" line that is being thrown around by IBM is crap. The entire reason for XML's existence is so that we can build different formats that satisfy different needs. After all, no one asks them why the ODF folks had to invent their own format when PDF and [X]HTML already exist. The fact that ODF and OOXML exist yet have different goals is fine. What is important is that they both are non-proprietary, open standards which prevents customers from being locked-in which is what people really want.

And I thought the RSS vs. Atom wars were pointless.

PS: On the issue of Wikipedia now using nofollow links, I kinda prefer Shelley Powers's idea in her post Wikipedia and nofollow that search engines treat Wikipedia specially as an 'instant answer' (MSN speak) or OneBox result (Google speak) instead of including it in the organic search results page. It has earned its place on the Web and should be treated specially including the placement of disclaimers warning Web n00bs that it's information should be taken with a grain of salt.

 

Joel Spolsky has an seminal article entitled Don't Let Architecture Astronauts Scare You where he wrote

A recent example illustrates this. Your typical architecture astronaut will take a fact like "Napster is a peer-to-peer service for downloading music" and ignore everything but the architecture, thinking it's interesting because it's peer to peer, completely missing the point that it's interesting because you can type the name of a song and listen to it right away.

All they'll talk about is peer-to-peer this, that, and the other thing. Suddenly you have peer-to-peer conferences, peer-to-peer venture capital funds, and even peer-to-peer backlash with the imbecile business journalists dripping with glee as they copy each other's stories: "Peer To Peer: Dead!"

 The Architecture Astronauts will say things like: "Can you imagine a program like Napster where you can download anything, not just songs?" Then they'll build applications like Groove that they think are more general than Napster, but which seem to have neglected that wee little feature that lets you type the name of a song and then listen to it -- the feature we wanted in the first place. Talk about missing the point. If Napster wasn't peer-to-peer but it did let you type the name of a song and then listen to it, it would have been just as popular.

This article is relevant because I recently wrote a series of posts explaining why Web developers have begun to favor JSON over XML in Web Services. My motivation for writing this article were conversations I'd had with former co-workers who seemed intent on "abstracting" the discussion and comparing whether JSON was a better data format than XML in all the cases that XML is used today instead of understanding the context in which JSON has become popular.

In the past two weeks, I've seen three different posts from various XML heavy hitters committing this very sin

1. JSON and XML by Tim Bray - This kicked it off and starts off by firing some easily refutable allegations about the extensibility and unicode capabilities of JSON as a general data transfer format.
   
2. Tim Bray on JSON and XML by Don Box - Refutes the allegations by Tim Bray above but still misses the point.
   
3. All markup ends up looking like XML by David Megginson - argues that XML is just like JSON except with the former we use angle brackets and in the latter we use curly braces + square brackets. Thus they are "Turing" equivalent. Academically interesting but not terribly useful information if you are a Web developer trying to get things done.
   

This is my plea to you, if you are an XML guru and you aren't sure why JSON seems to have come out of nowhere to threaten your precious XML, go read JSON vs. XML: Browser Security Model and JSON vs. XML: Browser Programming Models then let's have the discussion.

If you're too busy to read them, here's the executive summary. JSON is a better fit for Web services that power Web mashups and AJAX widgets due to the fact it gets around the cross domain limitations put in place by browsers that hamper XMLHttpRequest and that it is essentially serialized Javascript objects which makes it fit better client side scripting which is primarily done in Javascript. That's it. XML will never fit the bill as well for these scenarios without changes to the existing browser ecosystem which I doubt are forthcoming anytime soon.

Update: See comments by David Megginson and Steve Marx below.

 

Over the holidays I had a chance to talk to some of my old compadres from the XML team at Microsoft and we got to talking about the JSON as an alternative to XML. I concluded that there are a small number of key reasons that JSON is now more attractive than XML for kinds of data interchange that powers Web-based mashups and Web gadgets widgets. This is the second in a series of posts on what these key reasons are.

In my previous post, I mentioned that getting around limitations in cross domain requests imposed by modern browsers has been a key reason for the increased adoption of JSON. However this is only part of the story.

Early on in the adoption of AJAX techniques across various Windows Live services I noticed that even for building pages with no cross domain requirements, our Web developers favored JSON to XML. One response that kept coming up is the easier programming model when processing JSON responses on the client than with XML. I'll illustrate this difference in ease of use via a JScript code that shows how to process a sample document in both XML and JSON formats taken from the JSON website. Below is the code sample

var json_menu = '{"menu": {' + '\n' +
'"id": "file",' + '\n' +
'"value": "File",' + '\n' +
'"popup": {' + '\n' +
'"menuitem": [' + '\n' +
'{"value": "New", "onclick": "CreateNewDoc()"},' + '\n' +
'{"value": "Open", "onclick": "OpenDoc()"},' + '\n' +
'{"value": "Close", "onclick": "CloseDoc()"}' + '\n' +
']' + '\n' +
'}' + '\n' +
'}}';


var xml_menu = '<menu id="file" value="File">' + '\n' +
'<popup>' + '\n' +
'<menuitem value="New" onclick="CreateNewDoc()" />' + '\n' +
'<menuitem value="Open" onclick="OpenDoc()" />' + '\n' +
'<menuitem value="Close" onclick="CloseDoc()" />' + '\n' +
'</popup>' + '\n' +
'</menu>';

WhatHappensWhenYouClick_Xml(xml_menu);
WhatHappensWhenYouClick_Json(json_menu);

function WhatHappensWhenYouClick_Json(data){

  var j = eval("(" + data + ")");

  WScript.Echo("When you click the " + j.menu.value + " menu, you get the following options");

  for(var i = 0; i < j.menu.popup.menuitem.length; i++){
   WScript.Echo((i + 1) + "." + j.menu.popup.menuitem[i].value
    + " aka " + j.menu.popup.menuitem[i].onclick);
  }

}

function WhatHappensWhenYouClick_Xml(data){

  var x = new ActiveXObject( "Microsoft.XMLDOM" );
  x.loadXML(data);

  WScript.Echo("When you click the " + x.documentElement.getAttribute("value")
                + " menu, you get the following options");

  var nodes = x.documentElement.selectNodes("//menuitem");

  for(var i = 0; i < nodes.length; i++){
   WScript.Echo((i + 1) + "." + nodes[i].getAttribute("value") + " aka " + nodes[i].getAttribute("onclick"));
  }
}

When comparing both sample functions, it seems clear that the XML version takes more code and requires a layer of mental indirection as the developer has to be knowledgeable about XML APIs and their idiosyncracies. We should dig a little deeper into this. 

A couple of people have already replied to my previous post to point out that any good Web application should process JSON responses to ensure they are not malicious. This means my usage of eval() in the code sample, should be replaced with JSON parser that only accepts 'safe' JSON responses. Given that that there are JSON parsers available that come in under 2KB that particular security issue is not a deal breaker.

On the XML front, there is no off-the-shelf manner to get a programming model as straightforward and as flexible as that obtained from parsing JSON directly into objects using eval(). One light on the horizon is that E4X becomes widely implemented in Web browsers . With E4X, the code for processing the XML version of the menu document above would be 

function WhatHappensWhenYouClick_E4x(data){

  var e = new XML(data);

  WScript.Echo("When you click the " + j.menu.value + " menu, you get the following options");

  foreach(var m in e.menu.popup.menuitem){
   WScript.Echo( m.@value + " aka " + m.@onclick);
  }

}

However as cool as the language seems to be it is unclear whether E4X will ever see mainstream adoption. There is an initial implementation of E4X in the engine that powers the Firefox browser which seems to be incomplete. On the other hand, there is no indication that either Opera or Internet Explorer will support E4X in the future.

Another option for getting the simpler object-centric programming models out of XML data could be to adopt a simple XML serialization format such as XML-RPC and providing off-the-shelf Javascript parsers for this data format. A trivial implementation could be for the parser to convert XML-RPC to JSON using XSLT then eval() the results. However it is unlikely that people would go through that trouble when they can just use JSON.

This may be another nail in the coffin of XML on the Web. 

 

Over the holidays I had a chance to talk to some of my old compadres from the XML team at Microsoft and we got to talking about the JSON as an alternative to XML. I concluded that there are a small number of key reasons that JSON is now more attractive than XML for kinds of data interchange that powers Web-based mashups and Web gadgets widgets. This is the first in a series of posts on what these key reasons are.

The first "problem" that chosing JSON over XML as the output format for a Web service solves is that it works around security features built into modern browsers that prevent web pages from initiating certain classes of communication with web servers on domains other than the one hosting the page. This "problem" is accurately described in the XML.com article Fixing AJAX: XMLHttpRequest Considered Harmful which is excerpted below

But the kind of AJAX examples that you don't see very often (are there any?) are ones that access third-party web services, such as those from Amazon, Yahoo, Google, and eBay. That's because all the newest web browsers impose a significant security restriction on the use of XMLHttpRequest. That restriction is that you aren't allowed to make XMLHttpRequests to any server except the server where your web page came from. So, if your AJAX application is in the page http://www.yourserver.com/junk.html, then any XMLHttpRequest that comes from that page can only make a request to a web service using the domain www.yourserver.com. Too bad -- your application is on www.yourserver.com, but their web service is on webservices.amazon.com (for Amazon). The XMLHttpRequest will either fail or pop up warnings, depending on the browser you're using.

On Microsoft's IE 5 and 6, such requests are possible provided your browser security settings are low enough (though most users will still see a security warning that they have to accept before the request will proceed). On Firefox, Netscape, Safari, and the latest versions of Opera, the requests are denied. On Firefox, Netscape, and other Mozilla browsers, you can get your XMLHttpRequest to work by digitally signing your script, but the digital signature isn't compatible with IE, Safari, or other web browsers.

This restriction is a significant annoyance for Web developers because it eliminates a number of compelling end user applications due to the limitations it imposes on developers. However, there are a number of common workarounds which are also listed in the article

Solutions Worthy of Paranoia

There is hope, or rather, there are gruesome hacks, that can bring the splendor of seamless cross-browser XMLHttpRequests to your developer palette. The three methods currently in vogue are:

1. Application proxies. Write an application in your favorite programming language that sits on your server, responds to XMLHttpRequests from users, makes the web service call, and sends the data back to users.
2. Apache proxy. Adjust your Apache web server configuration so that XMLHttpRequests can be invisibly re-routed from your server to the target web service domain.
3. Script tag hack with application proxy (doesn't use XMLHttpRequest at all). Use the HTML script tag to make a request to an application proxy (see #1 above) that returns your data wrapped in JavaScript. This approach is also known as On-Demand JavaScript.

Although the first two approaches work, there are a number of problems with them. The first is that it adds a requirement that the owner of the page also have Web master level access to a Web server and either tweak its configuration settings or be a savvy enough programmer to write an application to proxy requests between a user's browser and the third part web service. A second problem is that it significantly increases the cost and scalability impact of the page because the Web page author now has to create a connection to the third party Web service for each user viewing their page instead of the user's browser making the connection. This can lead to a bottleneck especially if the page becomes popular. A final problem is that if the third party service requires authentication [via cookies] then there is no way to pass this information through the Web page author's proxy due to browser security models.

The third approach avoids all of these problems without a significant cost to either the Web page author or the provider of the Web service. An example of how this approach is utilized in practice is described in Simon Willison's post JSON and Yahoo!’s JavaScript APIs where he writes

As of today, JSON is supported as an alternative output format for nearly all of Yahoo!’s Web Service APIs. This is a Really Big Deal, because it makes Yahoo!’s APIs available to JavaScript running anywhere on the web without any of the normal problems caused by XMLHttpRequest’s cross domain security policy.

Like JSON itself, the workaround is simple. You can append two arguments to a Yahoo! REST Web Service call:

&output=json&callback=myFunction

The page returned by the service will look like this:

myFunction({ JSON data here });

You just need to define myFunction in your code and it will be called when the script is loaded. To make cross-domain requests, just dynamically create your script tags using the DOM:

var script = document.createElement('script');
script.type = 'text/javascript';
script.src = '...' + '&output=json&callback=myFunction';
document.getElementsByTagName('head')[0].appendChild(script);

People who are security minded will likely be shocked that this technique involves Web pages executing arbitrary code they retrieve from a third party site since this seems like a security flaw waiting to happen especially if the 3rd party site becomes compromised. One might also wonder what's the point of browsers restricting cross-domain HTTP requests if pages can load and run arbitrary Javascript code [not just XML data] from any domain.

However despite these concerns, it gets the job done with minimal cost to all parties involved and more often than not that is all that matters.

Postscript: When reading articles like Tim Bray's JSON and XML which primarily compares both data formats based on their physical qualities, it is good to keep the above information in mind since it explains a key reason JSON is popular on the Web today which turns out to be independent of any physical qualities of the data format. 

 

Mark Baker has a blog post entitled Validation considered harmful where he writes

We believe that virtually all forms of validation, as commonly practiced, are harmful; an anathema to use at Web scale. Specifically, our argument is this;

Tests of validity which are a function of time make the independent evolution of software problematic.

Why? Consider the scenario of two parties on the Web which want to exchange a certain kind of document. Party A has an expensive support contract with BigDocCo that ensures that they’re always running the latest-and-greatest document processing software. But party B doesn’t, and so typically lags a few months behind. During one of those lags, a new version of the schema is released which relaxes an earlier stanza in the schema which constrained a certain field to the values “1″, “2″, or “3″; “4″ is now a valid value. So, party B, with its new software, happily fires off a document to A as it often does, but this document includes the value “4″ in that field. What happens? Of course A rejects it; it’s an invalid document, and an alert is raised with the human adminstrator, dramatically increasing the cost of document exchange. All because evolvability wasn’t baked in, because a schema was used in its default mode of operation; to restrict rather than permit.

This doesn't seem like a very good argument to me. The fact that you enforce that the XML documents you receive must follow a certain structure or must conform to certain constraints does not mean that your system cannot be flexible in the face of new versions. First of all, every system does some form of validation because it cannot process arbitrary documents. For example an RSS reader cannot do anything reasonable with an XBRL or ODF document, no matter how liberal it is in what it accepts. Now that we have accepted that there are certain levels validation that are no-brainers the next question is to ask what happens if there are no constraints on the values of elements and attributes in an input document. Let's say we have a purchase order format which in v1 has a <currency> element which can have a value of "U.S. dollars" or "Canadian dollars" then in v2 we now support any valid currency. What happens if a v2 document is sent to a v1 client? Is it a good idea for such a client to muddle along even though it can't handle the specified currency format?

As in all things in software, there are no hard and fast rules as to what is right and what is wrong. In general, it is better to be flexible rather than not as the success of HTML and RSS have shown us but this does not mean that it is acceptable in every situation. And it comes with its own set of costs as the success of HTML and RSS have shown us. :)

Sam Ruby puts it more eloquently than I can in his blog post entitled Tolerance.

 

Edd Dumbill has a blog post entitled Afraid of the POX? where he writes

The other day I had was tinkering with that cute little poster child of Web 2.0, Flickr. Looking for a lightweight way to incorporate some photos into a web site, I headed to their feeds page to find some XML to use.
...
The result was interesting. Flickr have a variety of outputs in RSS dialects, but you just can't get at the raw data using XML. The bookmarking service del.icio.us is another case in point. My friend Matt Biddulph recently had to resort to screenscraping in order to write his tag stemmer, until some kind soul pointed out there's a JSON feed.

Both of these services support XML output, but only with the semantics crammed awkwardly into RSS or Atom. Neither have plain XML, but do support serialization via other formats. We don't really have "XML on the Web". We have RSS on the web, plus a bunch of mostly JSON and YAML for those who didn't care for pointy brackets.

Interesting set of conclusions but unfortunately based on faulty data. Flickr provides custom XML output from their Plain Old XML over HTTP APIs at http://www.flickr.com/services/api as does del.icio.us from its API at http://del.icio.us/help/api. If anything, this seems to indicate that old school XML heads like Edd have a different set of vocabulary from the Web developer crowd. It seems Edd did searches for "XML feeds" from these sites then came off irritated that the data was in RSS/Atom and not custom XML formats. However once you do a search for "API" with the appropriate service name, you find their POX/HTTP APIs which provide custom XML output.

The morale of this story is that "XML feeds" pretty much means RSS/Atom feeds these days and is not a generic term for XML being provided by a website.

PS: This should really be a comment on Edd's blog but it doesn't look like his blog supports comment.
 

If you are a reggular reader of Slashdot you probably stumbled on a link to the Groklaw article Novell "Forking" OpenOffice.org by Pamela Jones. In the article, she berates Novell for daring to provide support for the Office Open XML formats in their version of OpenOffice.

Miguel De Icaza, a Novell employee, has posted a response entitled OpenOffice Forks? where he writes

Facts barely matter when they get in the way of a good smear. The comments over at Groklaw are interesting, in that they explore new levels of ignorance.

Let me explain.

We have been working on OpenOffice.Org for longer than anyone else has. We were some of the earliest contributors to OpenOffice, and we are the largest external contributor to actual code to OpenOffice than anyone else.
...
Today we ship modified versions of OpenOffice to integrate GStreamer, 64-bit fixes, integrate with the GNOME and KDE file choosers, add SVG importing support, add OpenDMA support, add VBA support, integrate Mono, integrate fontconfig, fix bugs, improve performance and a myriad of others. The above url contains some of the patches that are pending, but like every other open source project, we have published all of those patches as part of the src.rpm files that we shipped, and those patches have eventually ended up in every distribution under the sun.

But the problem of course is not improving OpenOffice, the problem is improving OpenOffice in ways that PJ disapproves of. Improving OpenOffice to support an XML format created by Microsoft is tantamount to treason.

And of course, the code that we write to interop with Office XML is covered by the Microsoft Open Specification Promise (Update: this is a public patent agreement, this has nothing to do with the Microsoft/Novell agreement, and is available to anyone; If you still want to email me, read the previous link, and read it twice before hitting the send button).

I would reply to each individual point from PJ, but she either has not grasped how open source is actually delivered to people or she is using this as a rallying cry to advance her own ideological position on ODF vs OfficeXML.

Debating the technical merits of one of those might be interesting, but they are both standards that are here to stay, so from an adoption and support standpoint they are a no-brainer to me. The ideological argument on the other hand is a discussion as interesting as watching water boil. Am myself surprised at the spasms and epileptic seizures that folks are having over this.

I've been a fan of Miguel ever since I was a good lil' Slashbot in college. I've always admired his belief in "Free" [as in speech] Software and the impact it has on people's lives as well as the fact that he doesn't let geeky religious battles get in the way of shipping code. When Miguel saw good ideas in Microsoft's technologies, he incorporated the ideas into Bonobo and Mono as a way to improve the Linux software landscape instead of resorting to Not Invented Here syndrome.

Unfortunately, we don't have enough of that in the software industry today.

 

Tim Bray has a blog post entitled Choose RELAX Now where he writes

Elliotte Rusty Harold’s RELAX Wins may be a milestone in the life of XML. Everybody who actually touches the technology has known the truth for years, and it’s time to stop sweeping it under the rug. W3C XML Schemas (XSD) suck. They are hard to read, hard to write, hard to understand, have interoperability problems, and are unable to describe lots of things you want to do all the time in XML. Schemas based on Relax NG, also known as ISO Standard 19757, are easy to write, easy to read, are backed by a rigorous formalism for interoperability, and can describe immensely more different XML constructs. To Elliotte’s list of important XML applications that are RELAX-based, I’d add the Atom Syndication Format and, pretty soon now, the Atom Publishing Protocol. It’s a pity; when XSD came out people thought that since it came from the W3C, same as XML, it must be the way to go, and it got baked into a bunch of other technology before anyone really had a chance to think it over. So now lots of people say “Well, yeah, it sucks, but we’re stuck with it.” Wrong! The time has come to declare it a worthy but failed experiment, tear down the shaky towers with XSD in their foundation, and start using RELAX for all significant XML work.

In a past life I was the PM for XML schema technologies at Microsoft so I obviously have an opinion here. What Tim Bray and Elliotte Rusty Harold gloss over in their advocacy is that there are actually two reasons one would choose an XML schema technology. I covered both reasons in my article XML Schema Design Patterns: Is Complex Type Derivation Unnecessary? for XML.com a few years ago. The relevant part of the article is excerpted below

As usage of XML and XML schema languages has become more widespread, two primary usage scenarios have developed around XML document validation and XML schemas.

1. Describing and enforcing the contract between producers and consumers of XML documents: An XML schema ordinarily serves as a means for consumers and producers of XML to understand the structure of the document being consumed or produced. Schemas are a fairly terse and machine readable way to describe what constitutes a valid XML document according to a particular XML vocabulary. Thus a schema can be thought of as contract between the producer and consumer of an XML document. Typically the consumer ensures that the XML document being received from the producer conforms to the contract by validating the received document against the schema.
   
   This description covers a wide array of XML usage scenarios from business entities exchanging XML documents to applications that utilize XML configuration files.

2. Creating the basis for processing and storing typed data represented as XML documents: As XML became popular as a way to represent rigidly structured, strongly typed data, such as the content of a relational database or programming language objects, the ability to to describe the datatypes within an XML document became important. This led to Microsoft's XML Data and XML Data-Reduced schema languages, which ultimately led to WXS. These schema languages are used to convert an input XML infoset into a type annotated infoset (TAI) where element and attribute information items are annotated with a type name.
   
   WXS describes the creation of a type annotated infoset as a consequence of document validation against a schema. During validation against a WXS, an input XML infoset is converted into a post schema validation infoset (PSVI), which among other things contains type annotations. However practical experience has shown that one does not need to perform full document validation to create type annotated infosets; in general many applications that use XML schemas to create strongly typed XML such as XML<->object mapping technologies do not perform full document validation, since a number of WXS features do not map to concepts in the target domain.

RELAX NG is good at #1 but not #2 which is by design. Most of the folks who are interested in XSD are either WS-* folks who are building toolkits that map XML on the wire to in-memory objects or database folks implementing XQuery who also have to deal with strongly typed data. Neither category of developers/vendors are interested in RELAX NG because it wasn't designed to meet their needs. On the other hand, if you are designing an XML format from scratch and need a language/toolkit for validating the structure and correctness of your documents you definitely need to strongly consider using RELAX NG over XSD.

 

Brian Jones has a blog post entitled Politics behind standardization where he writes

We ultimately need to prioritize our standardization efforts, and as the Ecma Office Open XML spec is clearly further along in meeting the goal of full interoperability with the existing set of billions of Office documents, that is where our focus is. The Ecma spec is only a few months away from completion, while the OASIS committee has stated they believe they have at least another year before they are even able to define spreadsheet formulas. If the OASIS Open Document committee is having trouble meeting the goal of compatibility with the existing set of Office documents, then they should be able to leverage the work done by Ecma as the draft released back in the spring is already very detailed and the final draft should be published later this year.

To be clear, we have taken a 'hands off' approach to the OASIS technical committees because:  a) we have our hands full finishing a great product (Office 2007) and contributing to Ecma TC45, and b) we do not want in any way to be perceived as slowing down or working against ODF.  We have made this clear during the ISO consideration process as well.  The ODF and Open XML projects have legitimate differences of architecture, customer requirements and purpose.  This Translator project and others will prove that the formats can coexist with a certain tolerance, despite the differences and gaps.

No matter how well-intentioned our involvement might be with ODF, it would be perceived to be self-serving or detrimental to ODF and might come from a different perception of requirements.   We have nothing against the different ODF committees' work, but just recognize that our presence and input would tend to be misinterpreted and an inefficient use of valuable resources.  The Translator project we feel is a good productive 'middle ground' for practical interoperability concerns to be worked out in a transparent way for everyone, rather than attempting to swing one technical approach and set of customer requirements over to the other.

As someone who's watched standards committees from the Microsoft perspective while working on the XML team, I agree with everything Brian writes in his post. Trying to merge a bunch of contradictory requirements often results in a complex technology that causes more problems than it solves (e.g. W3C XML Schema). In addition, Microsoft showing up and trying to change the direction of the project to supports its primary requirement (an XML file format compatible with the legacy Microsoft Office file formats) would not be well received.

Unfortunately, the ODF discussion has seemed to be more political than technical which often obscures the truth. Microsoft is making moves to ensure that Microsoft Office not only provides the best features for its customers but ensures that they can exchange documents in a variety of document formats from those owned by Microsoft to PDF and ODF. I've seen a lot of customers acknowledge this truth and commend the company for it. At the end of the day, that matters a lot more than what competitors and detractors say. Making our customers happy is job #1. 

 

The Office team continues to impress me how savvy they are about the changing software landscape. In his blog post entitled Open XML Translator project announced (ODF support for Office) Brian Jones writes

Today we are announcing the creation of the Open XML Translator project that will help translate between the Office Open XML formats and the OpenDocument format. We've talked a lot about the value the Open XML formats bring, and one of them of course is the ability to filter it down into other formats. While we still aren't seeing a strong demand for ODF support from our corporate or consumer customers, it's now a bit different with governments. We've had some governments request that we help build solutions so that can use ODF for certain situations, so that's why we are creating the Open XML Translator project. I think it's going to be really beneficial to a number of folks and for a number of reasons.

There has been a push in Microsoft for better interoperability and this is another great step in that direction. We already have the PDF and XPS support for Office 2007 users that unfortunately had to be separated out of the product and instead offered as a free download. There will be a menu item in the Office applications that will point people to the downloads for XPS, PDF, and now ODF. So you'll have the ability to save to and open ODF files directly within Office (just like any other format).

For me, one of the really cool parts of this project is that it will be open source and located up on SourceForge, which means everyone will have the ability to see how to leverage the open architectures of both the Office Open XML formats and ODF. We're developing the tools with the help of Clever Age (based in France) and a few other folks like Aztecsoft (based in India) and Dialogika (based in Germany). There should actually be a prototype of the first translator (for Word 2007) posted up on SourceForge later on today (http://sourceforge.net/projects/odf-converter). It's going to be made available under the BSD license, and anyone can provide feedback, submit bugs, and of course directly contribute to the project. The Word tool should be available by the end of this year, with the Excel and PPT versions following in 2007.

This announcement is cool on so many levels. The coolest being that the projects will not only be Open Source but will be hosted on SourceForge. That is sweet. It is interesting to note that it is government customers and not businesses that are interested in ODF support in Office. I guess that makes sense if you consider which parties have been expressing interest in Open Office.

There already some great analyst responses to this move such as Stephen O'Grady of Redmonk who in his post Microsoft Office to Support ODF: The Q&A has some great insights. My favorite insight is excerpted below

Q: How about Microsoft's competitors?
A: Well, this is a bittersweet moment for them. For those like Corel that have eschewed ODF support, it's a matter of minor importance - at least until Microsoft is able to compete in public sector markets that mandate ODF and they are not.

But for those vendors that have touted ODF support as a diffentiator, this is a good news/bad news deal. The good news is that they can and almost certainly will point to Microsoft's support as validation of further ODF traction and momentum, they will now be competing - at least in theory, remember the limitation - with an Office suite that is frankly the most capable on the market. I've said for years that packages like OpenOffice.org are more than good enough for the majority of users, and that's been validated by our own usage of the product over the past few years; but Microsoft's suite is better than good enough. I'm interested to see if there's any fallout from the UI overhaul, but for now Office remains the undisputed champ of the Office arena. This means that commercial packages like StarOffice and Workplace, not to mention open source projects such as Abiword, KOffice, and OpenOffice.org will have to compete more on features and innovation and less on their support for formats such as ODF or PDF.

It'll be good to see the debate migrate away from support for file formats back to exactly which product's features provides the best value for customers. Everybody wins. Mad props to the Office team for making this decision. Rock on.

 

Mike Champion has a blog post entitled Why does the world need another XML API? where he writes

One basic question keeps coming up, something like: "We have SAX, DOM, XmlReader/Writer APIs (and the Java people have a bunch more), we have XSLT, we have XQuery ... why do you think we need Yet Another XML API?"
...

• XmlReader / XmlWriter can't go away because XLinq uses them to parse and serialize between XLinq objects and XML text. Also, while we are making XLinq as streaming-friendly as possible (see the XStreamingElement class in the CTP release for a taste of where we are going), we're only aiming at hitting the 80/20 point...
• DOM can't go away because there are important use cases for API-level interoperability, most notably in the browser...DOM doesn't make code truly interoperable across implementations (especially on other languages), but there is enough conceptual similarity that porting is generally not terribly difficult...  
• XSLT definitely won't go away. The Microsoft XML team was promoting XQuery as a "better XSLT than XSLT 2.0" a few years ago (before I came, don't hurt me!), and got set straight by the large and vocal XSLT user community on why this is not going to fly. While it may be true in some abstract way that XQuery or XLinq might logically be able to do everything that XSLT does, as a practical matter it won't...  
  
• XQuery won't go away, at least for its original use case as a database query language.  Microsoft supports a draft of XQuery in SQL Server 2005, contributes to the work of the XQuery working group at W3C, and will continue to invest in finalizing the XQuery Recommendation and implementing it in our DBMS..
  

we believe that the overall LINQ story is going to have a pretty profound impact on data programmability, and we want to make sure that LINQ has a good story for XML...For XML users, I see a few really powerful implications:

• The ability to query data by declaraing the characterics of the result set rather than imperatively navigating through and filtering out all the data...
• The ability to join across diverse data sources, be they XML documents, objects, or DBMS queries
• The ability to "functionally" reshape data within the same language as the application is written.  XSLT pioneered the functional transformation approach to XML processing, but it is difficult for many developers to learn and requires a processing pipeline architecture to combine XSLT transforms with conventional application logic...

This brings back memories of my days on the XML team at Microsoft. We went back and forth a lot about building the "perfect XML API", the one problem we had was that there one too many diverse user bases which had different ideas of what was important to expose in an API. We were always caught between a rock and a hard place when it came to customer requests for fixing our APIs. To some people (e.g. Microsoft Office) XML was a markup format for documents while to others (e.g. Windows Communications Foundation aka Indigo) it was simply a serialization format for programming language objects. Some of our customers were primarily interested in processing XML in a streaming fashion (e.g. Biztalk) while others (e.g. Internet Explorer) always worked on in-memory XML documents. Then there were the teams whose primarily interest was in strongly typed XML (e.g. SQL Server, ADO.NET) since it would be stored in relational database columns.

In trying to solve all of these problems with a single set of APIs, we went down the road of prematurely declaring the death of certain XML APIs and technologies such as the DOM (see Ode to the XML DOM) and XSLT (see XSLT 2.0 Sir? or Why You Won't See XSLT 2.0 or XPath 2.0 in the Next Version of the .NET Framework). At the end of the day we saw the light and we eventually changed our tune by not deprecating the System.Xml.XmlDocument class and by reconsidering whether replacing XSLT with XQuery was the right way forward.

When I was on the team there was a lot of infatuation with XQuery which eventually turned to frustration. There were a number of technical and non-technical reasons for this such as its dependence on W3C XML Schema which significantly complicated its type system and how long the spec was taking to become a standard (over 5 years and counting as I write this post). Since then a bunch of folks who were were enamored with XQuery have taken some of its lessons (e.g. declaritiveness, simple model for XML generation, etc) and integrated it into a mainstream programming environment with the XLinq project. XML geeks everywhere should read Erik Meijer's paper, XLinq: XML Programming Refactored (The Return Of The Monoids), it is a thing of beauty if angle brackets are your thing. And even better, if you are one you are one of those that chants rabid slogans like "XML is the assembly language of Web 2.0", you'll still like XLinq because it provides a easier and richer level of abstraction for working with XML.

Enjoy.

 

If you're a regular reader of Don Box's weblog then you probably know that Microsoft has made available another Community Technical Preview (CTP) of Language Integrated Query (LINQ) aka C# 3.0. I think the notion of integrating data access and query languages into programming languages is the next natural evolution in programming language design. A large number of developers write code that performs queries over rich data structures of some sort whether they are relational databases, XML files or just plain old objects in memory. In all three cases, the code tends to be verbose and more cumbersome than it needs to be. The goal of the LINQ project is to try to simplify and unify data access in programming languages built on the .NET Framework. 

When I used to work on the XML team, we also used to salivate about the power that developers would get if they could get rich query over their data stores in a consistent manner. I was the PM for the IXPathNavigable interface and the related XPathNavigator class which we hoped people would implement over their custom stores to enable them to use XPath to query them. Some developers did do exactly that such as Steve Saxon with the ObjectXPathNavigator which allows you to use XPath to query a graph of in-memory objects. The main problem with this approach is that implementing IXPathNavigable for custom data stores is non-trivial especially given the impedence mismatch between XML and other data models. In fact, I've been wanting to do something like this in RSS Bandit for a while but the complexity of implementing my own custom XPathNavigator class over our internal data structures is something I've balked at doing.

According to Matt Warren's blog post Oops, we did it again it looks like the LINQ folks have similar ideas but are making it easier than we did on the XML team. He writes 

What's the coolest new feature?  IMHO, its IQueryable<T>. 

 DLINQ's query mechanism has been generalized and available for all to use as part of System.Query.  It implements the Standard Query Operators for you using expression nodes to represent the query. Your queries can now be truly polymorphic, written over a common abstraction and translated into the target environment only when you need it to.

    public int CustomersInLondon(IQueryable<Customer> customers) {

        int count = (from c in customers

                     where c.City == "London"

                     select c).Count();

        return count;

    }

Now you can define a function like this and it can operate on either an in memory collection or a remote DLINQ collection (or you own IQueryable for that matter.)  The query is then either run entirely locally or remotely depending on the target. 

If its a DLINQ query a count query is sent to the database.

SELECT COUNT(*) AS [value]

FROM [Customers] AS [t0]

WHERE [t0].[City] = @p0

If its a normal CLR collection, the query is executed locally, using the System.Query.Sequence classes definitions of the standard query operators.  All you need to do is turn your IEnumerable<Customer> into IQueryable<Customer>.  This is accomplished easily with a built-in ToQueryable() method.

  List<Customer> customers = ...;

  CustomersInLondon(customers.ToQueryable());

Wow!  That was easy.  But, how is this done?  How can you possible turn my List<T> into some queryable thingamabob?

Good question.  Glad you asked.

Check out this little gem: 

  Expression<Func<Customer,bool>> predicate = c => c.City == "London";           

  Func<Customer,bool> d = predicate.Compile();

 

Now you can compile lambda expressions directly into IL at runtime!

ToQueryable() wraps your IEnumerable<T> in IQueryable<T> clothing, uses the Queryable infrastructure to let you build up your own expression tree queries, and then when you enumerate it, the expression is rebound to refer to your IEnumerable<T> directly, the operators rebound to refer to System.Query.Sequence, and the resulting code is compiled using the built-in expression compiler.  That code is then invoked producing your results.

Amazing, but true.

I think it's pretty amazing that all I have to do as a developer is implement a simple iterator over my data structures (i.e. IEnumerable) and then I get all the power of Linq for free. Of course, if I want the queries to be performant it would make sense to implement IQueryable directly but the fact that the barrier to entry is so low if my perf needs aren't high is goodness.

For more information on LINQ, read the Linq project overview. If you are like me and are primarily interested in XLinq then check out XLinq: XML Programming Refactored (The Return Of The Monoids) which has the fingerprints of my former team all over it. Way to go guys!

 

I just noticed that last week the W3C published a working draft specification for The XMLHttpRequest Object. I found the end of the working draft somewhat interesting. Read through the list of references and authors of the specifcation below

References

This section is normative

DOM3

Document Object Model (DOM) Level 3 Core Specification, Arnaud Le Hors (IBM), Philippe Le Hégaret (W3C), Lauren Wood (SoftQuad, Inc.), Gavin Nicol (Inso EPS), Jonathan Robie (Texcel Research and Software AG), Mike Champion (Arbortext and Software AG), and Steve Byrne (JavaSoft).

RFC2119

Key words for use in RFCs to Indicate Requirement Levels, S. Bradner.

RFC2616

Hypertext Transfer Protocol -- HTTP/1.1, R. Fielding (UC Irvine), J. Gettys (Compaq/W3C), J. Mogul (Compaq), H. Frystyk (W3C/MIT), L. Masinter (Xerox), P. Leach (Microsoft), and T. Berners-Lee (W3C/MIT).

B. Authors

This section is informative

The authors of this document are the members of the W3C Web APIs Working Group.

• Robin Berjon, Expway (Working Group Chair)
• Ian Davis, Talis Information Limited
• Gorm Haug Eriksen, Opera Software
• Marc Hadley, Sun Microsystems
• Scott Hayman, Research In Motion
• Ian Hickson, Google
• Björn Höhrmann, Invited Expert
• Dean Jackson, W3C
• Christophe Jolif, ILOG
• Luca Mascaro, HTML Writers Guild
• Charles McCathieNevile, Opera Software
• T.V. Raman, Google
• Arun Ranganathan, AOL
• John Robinson, AOL
• Doug Schepers, Vectoreal
• Michael Shenfield, Research In Motion
• Jonas Sicking, Mozilla Foundation
• Stéphane Sire, IntuiLab
• Maciej Stachowiak, Apple Computer
• Anne van Kesteren, Opera Software

Thanks to all those who have helped to improve this specification by sending suggestions and corrections. (Please, keep bugging us with your issues!)

Interesting. A W3C specification that documents a proprietary Microsoft API which not only does not include a Microsoft employee as a spec author but doesn't even reference any of the IXMLHttpRequest documentation on MSDN.

I'm sure there's a lesson in there somewhere. ;)

 

Brian Jones has a blog post entitled Corel to support Microsoft Office Open XML Formats which begins

Corel has stated that they will support the new XML formats in Wordperfect once we release Office '12'. We've already seen other applications like OpenOffice and Apple's TextEdit support the XML formats that we built in Office 2003. Now as we start providing the documentation around the new formats and move through Ecma we'll see more and more people come on board and support these new formats. Here is a quote from Jason Larock of Corel talking about the formats they are looking to support in coming versions (http://labs.pcw.co.uk/2006/01/new_wordperfect_1.html):

Larock said no product could match Wordperfect's support for a wide variety of formats and Corel would include OpenXML when Office 12 is released. "We work with Microsoft now and we will continue to work with Microsoft, which owns 90 percent of the market. We would basically cut ouirselves off if you didn't support the format."

But he admitted that X3 does not support the Open Document Format (ODF), which is being proposed as a rival standard, "because no customer that we are currently dealing with as asked us to do so."

X3 does however allow the import and export of portable document format (pdf) files, something Microsoft has promised for Office 12.

I mention this article because I wanted to again stress that even our competitors will now have clear documentation that allows them to read and write our formats. That isn't really as big of a deal though as the fact that any solution provider can do this. It means that the documents can now be easily accessed 100 years from now, and start to play a more meaningful role in business processes.

Again I want to extend my kudos to Brian and the rest of the folks on the Office team who have been instrumental in the transition of the Microsoft Office file formats from proprietary binary formats to open XML formats.

 

One part of the XML vision that has always resonated with me is that it encourages people to build custom XML formats specific to their needs but allows them to map between languages using technologies like XSLT. However XML technologies like XSLT focus on mapping one kind of syntax for another. There is another school of thought from proponents of Semantic Web technologies like RDF, OWL, and DAML+OIL, etc that higher level mapping between the semantics of languages is a better approach. 

In previous posts such as RDF, The Semantic Web and Perpetual Motion Machines and More on RDF, The Semantic Web and Perpetual Motion Machines I've disagreed with the thinking of Semantic Web proponents because in the real world you have to mess with both syntactical mappings and semantic mappings. A great example of this is shown in the post entitled On the Quality of Metadata... by Stefano Mazzocchi where he writes

One thing we figured out a while ago is that merging two (or more) datasets with high quality metadata results in a new dataset with much lower quality metadata. The "measure" of this quality is just subjective and perceptual, but it's a constant thing: everytime we showed this to people that cared about the data more than the software we were writing, they could not understand why we were so excited about such a system, where clearly the data was so much poorer than what they were expecting.

We use the usual "this is just a prototype and the data mappings were done without much thinking" kind of excuse, just to calm them down, but now that I'm tasked to "do it better this time", I'm starting to feel a little weird because it might well be that we hit a general rule, one that is not a function on how much thinking you put in the data mappings or ontology crosswalks, and talking to Ben helped me understand why.

First, let's start noting that there is no practical and objective definition of metadata quality, yet there are patterns that do emerge. For example, at the most superficial level, coherence is considered a sign of good care and (here all the metadata lovers would agree) good care is what it takes for metadata to be good. Therefore, lack of coherence indicates lack of good care, which automatically resolves in bad metadata.

Note how the is nothing but a syllogism, yet, it's something that, rationally or not, comes up all the time.

This is very important. Why? Well, suppose you have two metadatasets, each of them very coherent and well polished about, say, music. The first encodes Artist names as "Beatles, The" or "Lennon, John", while the second encodes them as "The Beatles" and "John Lennon". Both datasets, independently, are very coherent: there is only one way to spell an artist/band name, but when the two are merged and the ontology crosswalk/map is done (either implicitly or explicitly), the result is that some songs will now be associated with "Beatles, The" and others with "The Beatles".

The result of merging two high quality datasets is, in general, another dataset with a higher "quantity" but a lower "quality" and, as you can see, the ontological crosswalks or mappings were done "right", where for "right" I mean that both sides of the ontological equation would have approved that "The Beatles" or "Beatles, The" are the band name that is associated with that song.

At this point, the fellow semantic web developers would say "pfff, of course you are running into trouble, you haven't used the same URI" and the fellow librarians would say "pff, of course, you haven't mapped them to a controlled vocabulary of artist names, what did you expect?".. deep inside, they are saying the same thing: you need to further link your metadata references "The Beatles" or "Beatles, The" to a common, hopefully globally unique identifier. The librarian shakes the semantic web advocate's hand, nodding vehemently and they are happy campers.

The problem Stefano has pointed out is that just being able to say that two items are semantically identical (i.e. an artist field in dataset A is the same as the 'band name' field in dataset B) doesn't mean you won't have to do some syntactic mapping as well (i.e. alter artist names of the form "ArtistName, The" to "The ArtistName") if you want an accurate mapping.

The example I tend to cull from in my personal experience is mapping between different XML syndication formats such as Atom 1.0 and RSS 2.0. Mapping between both formats isn't simply a case of saying <atom:published>  owl:sameAs <pubDate> or that <atom:author>  owl:sameAs <author> . In both cases, an application that understands how to process one format (e.g. an RSS 2.0 parser) would not be able to process the syntax of the equivalent  elements in the other (e.g. processing RFC 3339 dates as opposed to RFC 822 dates).

Proponents of Semantic Web technologies tend to gloss over these harsh realities of mapping between vocabularies in the real world. I've seen some claims that simply using XML technologies for mapping between XML vocabularies means you will need N² transforms as opposed to needing 2N transforms if using SW technologies (Stefano mentions this in his post as has Ken Macleod in his post XML vs. RDF :: N × M vs. N + M). The explicit assumption here is that these vocabularies have similar data models and semantics which should be true otherwise a mapping wouldn't be possible. However the implicit assumption is that the syntax of each vocabulary is practically identical (e.g. same naming conventions, same date formats, etc) which this post provides a few examples where this is not the case. 

What I'd be interested in seeing is whether there is a way to get some of the benefits of Semantic Web technologies while acknowledging the need for syntactical mappings as well. Perhaps some weird hybrid of OWL and XSLT? One can only dream...

 

Since writing my post Microformats vs. XML: Was the XML Vision Wrong?, I've come across some more food for thought in the appropriateness of using microformats over XML formats. The real-world test case I use when thinking about choosing microformats over XML is whether instead of having an HTML web page for my blog and an Atom/RSS feed, I should instead have a single HTML  page with <div class="rss:item"> or <h3 class="atom:title"> embedded in it. To me this seems like a gross hack but I've seen lots of people comment on how this seems like a great idea to them. Given that I hadn't encountered universal disdain for this idea, I decided to explore further and look for technical arguments for and against both approaches.

I found quite a few discussions on the how and why microformats came about in articles such as The Microformats Primer in the Digital Web Magazine and Introduction to Microformats in the Microformats wiki. However I hadn't seen many in-depth technical arguments of why they were better than XML formats until recently. 

In a comment in response to my Microformats vs. XML: Was the XML Vision Wrong?, Mark Pilgrim wrote

Before microformats had a home page, a blog, a wiki, a charismatic leader, and a cool name, I was against using XHTML for syndication for a number of reasons.

http://diveintomark.org/archives/2002/11/26/syndication_is_not_publication

I had several basic arguments:

1. XHTML-based syndication required well-formed semantic XHTML with a particular structure, and was therefore doomed to failure. My experience in the last 3+ years with both feed parsing and microformats parsing has convinced me that this was incredibly naive on my part. Microformats may be *easier* to accomplish with semantic XHTML (just like accessibility is easier in many ways if you're using XHTML + CSS), but you can be embed structured data in really awful existing HTML markup, without migrating to "semantic XHTML" at all.

2. Bandwidth. Feeds are generally smaller than their corresponding HTML pages (even full content feeds), because they don't contain any of the extra fluff that people put on web pages (headers, footers, blogrolls, etc.) And feeds only change when actual content changes, whereas web pages can change for any number of valid reasons that don't involve changes to the content a feed consumer would be interested in. This is still valid, and I don't see it going away anytime soon.

3. The full-vs-partial content debate. Lots of people who publish full content on web pages (including their home page) want to publish only partial content in feeds. The rise of spam blogs that automatedly steal content from full-content feeds and republish them (with ads) has only intensified this debate.

4. Edge cases. Hand-crafted feed summaries. Dates in Latin. Feed-only content. I think these can be handled by microformats or successfully ignored. For example, machine-readable dates can be encoded in the title attribute of the human-readable date. Hand-crafted summaries can be published on web pages and marked up appropriately. Feed-only content can just be ignored; few people do it and it goes against one of the core microformats principles that I now agree with: if it's not human-readable in a browser, it's worthless or will become worthless (out of sync) over time.

I tend to agree with Mark's conclusions. The main issue with using microformats for syndication instead of RSS/Atom feeds is wasted bandwidth since web pages tend to contain more stuff than feeds and change more often.

Norm Walsh raises a few other good points on the trade offs being made when choosing microformats over XML in his post Supporting Microformats where he writes

Microformats (and architectural forms, and all the other names under which this technique has been invented) take this one step further by standardizing some of these attribute values and possibly even some combination of element types and attribute values in one or more content models.

This technique has some stellar advantages: it's relatively easy to explain and the fallback is natural and obvious, new code can be written to use this “extra” information without any change being required to existing applications, they just ignore it.

Despite how compelling those advantages are, there are some pretty serious drawbacks associated with microformats as well. Adding hCalendar support to my itineraries page reinforced several of them.

1. They're not very flexible. While I was able to add hCalendar to the overall itinerary page, I can't add it to the individual pages because they don't use the right markup. I'm not using <div> and <span> to markup the individual appointments, so I can't add hCalendar to them.
   

2. I don't think they'll scale very well. Microformats rely on the existing extensibility point, the role or class attribute. As such, they consume that extensibility point, leaving me without one for any other use I may have.
   

3. They're devilishly hard to validate. DTDs and W3C XML Schema are right out the door for validating microformats. Of course, Schematron (and other rule-based validation languages) can do it, but most of us are used to using grammar-based validation on a daily basis and we're likely to forget the extra step of running Schematron validation.
   

   It's interesting that RELAX NG can almost, but not quite, do it. RELAX NG has no difficulty distinguishing between two patterns based on an attribute value, but you can't use those two patterns in an interleave pattern. So the general case, where you want to say that the content of one of these special elements is “an <abbr> with class="dtstart" interleaved with an <abbr> with class="dtend" interleaved with…”, you're out of luck. If you can limit the content to something that doesn't require interleaving, you can use RELAX NG for your particular application, but most of the microformats I've seen use interleaving in the general case.
   

   Is validation really important? Well, I have well over a decade of experience with markup languages at this point and I was reminded just last week that I can't be relied upon to write a simple HTML document without markup errors if I don't validate it. If they can't be validated, they will often be incorrect.
   

The complexity of validating microformats isn't something I'd considered in my original investigation but is a valid point. As a developer of an RSS aggregator, I've found the existence of the Feed Validator to be an immense help in tracking down issues. Not having the luxury of being able to validate feeds would make building an aggregator a lot harder and a lot less fun. 

I'll continue to pay attention to this discussion but for now microformats will remain in the "gross hack" bucket for me.

 

Over a year ago, I wrote a blog post entitled SGML on the Web: A Failed Dream? where I asked whether the original vision of XML had failed. Below are excerpts from that post

The people who got together to produce the XML 1.0 recommendation where motivated to do this because they saw a need for SGML on the Web. Specifically  

their discussions focused on two general areas:

• Classes of software applications for which HTML was an inadequate information format
• Aspects of the SGML standard itself that impeded SGML's acceptance as a widespread information technology

The first discussion established the need for SGML on the web. By articulating worthwhile, even mission-critical work that could be done on the web if there were a suitable information format, the SGML experts hoped to justify SGML on the web with some compelling business cases.

The second discussion raised the thornier issue of how to "fix" SGML so that it was suitable for the web.

And thus XML was born.
...
The W3C's attempts to get people to author XML directly on the Web have mostly failed as can be seen by the dismal adoption rate of XHTML and in fact many [including myself] have come to the conclusion that the costs of adopting XHTML compared to the benefits are too low if not non-existent. There was once an expectation that content producers would be able to place documents conformant to their own XML vocabularies on the Web and then display would entirely be handled by stylesheets but this is yet to become widespread. In fact, at least one member of a W3C working group has called this a bad practice since it means that User Agents that aren't sophisticated enough to understand style sheets are left out in the cold.

Interestingly enough although XML has not been as successfully as its originators initially expected as a markup language for authoring documents on the Web it has found significant success as the successor to the Comma Separated Value (CSV) File Format. XML's primary usage on the Web and even within internal networks is for exchanging machine generated, structured data between applications. Speculatively, the largest usage of XML on the Web today is RSS and it conforms to this pattern.

These thoughts were recently rekindled when reading Tim Bray's recent post Don’t Invent XML Languages where Tim Bray argues that people should stop designing new XML formats. For designing new data formats for the Web, Tim Bray advocates the use of Microformats instead of XML.

The vision behind microformats is completely different from the XML vision. The original XML inventers started with the premise that HTML is not expressive enough to describe every possible document type that would be exchanged on the Web. Proponents of microformats argue that one can embed additional semantics over HTML and thus HTML is expressive enough to represent every possible document type that could be exchanged on the Web. I've always considered it a gross hack to think that instead of having an HTML web page for my blog and an Atom/RSS feed, instead I should have a single HTML  page with <div class="rss:item"> or <h3 class="atom:title"> embedded in it instead. However given that one of the inventors of XML (Tim Bray) is now advocating this approach, I wonder if I'm simply clinging to old ways and have become the kind of intellectual dinosaur I bemoan. 

 

Don Demsak has a post entitled XSLT 2.0, Microsoft, and the future of System.Xml which has some insightful perspectives on the future of XML in the .NET Framework

Oleg accidentally restarted the XSLT 2.0 on .Net firestorm by trying to startup an informal survey.  Dare chimed in with his view of how to get XSLT 2.0 in .Net.  M. David (the guy behind Saxon.Net which let .Net developers use Saxon on .Net) jumped in with his opinion.
...

One of the things that I’ve struggled with in System.Xml is how hard it is sometimes to extend the core library.  The XML MVPs have done a good job with some things, but other things (like implementing XSLT 2.0 on top of the XSLT 1.0 stuff) are impossible because so much of the library is buried in internal classes.  When building a complex library like System.Xml, there are 2 competing schools of thought:

1. Make the library easy to use and create a very small public facing surface area.
2. Make the library more of a core library with most classes and attributes public, and let others build easy (and very specific) object models on top of it.

The upside of the first methodology is that it is much easier to test, and the library just works out of the box.  The downside is that it very hard to extend the library, so it can only be used in very specific ways.

The upside of the second methodology is that you don’t have to trying to envision all the ways the library should be used.  Over time others will extend it to accomplish things that the original developers never thought of.  The downside is that you have a much larger surface area to test, and you are totally reliant on other projects to make your library useful.  This goes for both projects internal to Microsoft and external projects like the Mvp.Xml lib.

The System.Xml team has tended to use the first methodology, where the ASP.Net team tends to build their core stuff according to the second methodology, and then have a sub-team create another library using the first methodology, so developers have something to use right out of the box (think of System.Web.UI.HtmlControls as the low level API and System.Web.UI.WebControls as the higher level API).  The ASP.Net team builds their API this way because, from the beginning, they have always envisioned 3rd parties extending their library.  At the moment, this is not the case for the System.Xml library.  But the question is, should System.Xml be revamped and become a lower level API, and then rely on 3rd parties (like the Mvp.Xml project) to create more specific and easier to use APIs?  Obviously this is not something to be taken lightly.  It will be more costly to expose more of the internals of System.Xml.  But, if only the lower level API was part of the core .Net framework, it may then be possible to roll out newer, higher level, APIs on a release schedule different then the .Net framework schedule.  This way projects like XSLT 2.0 could be released without having to what for the next version of the framework.

 I’ve always been of the opinion that XSLT 2.0 does not need to be part of the core .Net framework.  Oleg doesn’t believe that the .Net open source community is as passionate as some of the other communities, so he would like to see Microsoft build XSLT 2.0.  I’d rather see the transformation of the System.Xml team into more of an ASP.Net like team.  If .Net is the future of development on the Windows platform, and XML is the future of Microsoft, then the System.Xml team needs to grow beyond its legacy as just an offshoot of the SQL Server team.  The System.Xml team still resides in the SQL Server building.  Back before .Net, the System.Xml was known as the SQL Web Data team, and unfortunately, still carries some of that mentality.  Folks like Joshua Allen and Dare (who are both not on the team anymore) fought to bring the team out from the shadows of SQL Server.  With new XML related groups, like XLinq and Windows Communication Framework, popping up within the company the System.Xml group is at a major crossroads.  They will either grow (in status and budget) and become more like the ASP.Net or they will get absorbed into one of the new groups.

 I’d prefer to see the System.Xml team grow and become full partners with teams like ASP.Net and the CLR team.  I’d like to see the XML based languages become first class programming languages within the Visual Studio IDE.  That means not only using things like XSLT and XML Schema as dynamic languages, but also be able to compile them down to IL and compiled with the other .Net languages.  I want to be able to create projects that contain not only VB or C#, but also XSLT and XML Schema (to name a couple), and have them compile into one executable.  Then developers can use things like XSLT 2.0, or the next in vogue XML based language, and take advantage of that language’s unique benefits, without having to choose between a compiled procedural language (like C# or VB) and dynamic functional languages like XSLT.  Linq is starting to bring in more of the functional programming style to the average procedural programmer, so I can start to see the rise public awareness of functional programming.  It is only a matter of time before the average programmer feels as comfortable with functional programming as they do with procedural programming, so we need to look towards including these languages within the Visual Studio IDE (which then leads into my discussion about evolving Visual Studio into more of an IDE Framework, and extended with add-ins.)

There is a lot of stuff which I agree with in Don's post which is why I forwarded it to some of the folks on the XML team. I'll be having lunch over there today to talk about some of the topics it raised

Don does gloss over something when it comes to the decision between whether Microsoft should implement a technology like XSLT 2.0 or whether we should just make it easy for third parties to do so. The truth is that Microsoft now has a large number of products which utilize XML-related technologies. For example, implementing something like XSLT 2.0 isn't just about providing a new version of the System.Xml.Xsl.XslCompiledTransform Class in the .NET Framework. It's also about deciding whether to update the XSLT engine used by Internet Explorer to support XSLT 2.0 (which is an entirely different code base), it's about adding support to the XSLT debugger in Visual Studio to support XSLT 2.0, and maybe even updating the Biztalk Mapper. Users of Microsoft products expect a cohesive and comprehensive experience. In many cases, only Microsoft can provide that experience (e.g. supporting XSLT 2.0 across our entire breadth of technologies and products that use XSLT). It was a really tough balance deciding what we should make extensible and what was too expensive to make extensible since we'd probably be the only ones who could take proper advantage of it when I was on the XML team. I'm glad to see that some of our MVPs understand how delicate of a balancing act shipping platform technologies can be.

 

I've been following a series of posts on Oleg Tkachenko's blog with some bemusement. In his post A business case for XSLT 2.0? he writes

If you are using XSLT and you think that XSLT 2.0 would provide you some real benefits, please drop a line of comment with a short explanation pleeeease. I'm collecting some arguments for XSLT 2.0, some real world scenarios that are hard with XSLT 1.0, some business cases when XSLT 2.0 would provide an additional value. That's really important if we want to have more than a single XSLT 2.0 implementation...

PS. Of course I've read Kurt's "The Business Case for XSLT 2.0" already.

Update: I failed to stress it enough that it's not me who needs such kind of arguments. We have sort of unique chance to persuade one of software giants (guess which one) to support XSLT 2.0 now.

In a follow up post entitled XSLT 2.0 and Microsoft Unofficial Survey he reveals which of the software giants he is trying to convince to implement XSLT 2.0 where he writes

Moving along business cases Microsoft seeks to implement XSLT 2.0 I'm trying to gather some opinion statistics amongs developers working with XML and XSLT. So I'm holding this survey at the XML Lab site:

Would you like to have XSLT 2.0 implementation in the .NET Framework?

The possible answers are:

• Yes, I need XSLT 2.0
• Yes, that would be nice to have
• No, continue improving XSLT 1.0 impl instead
• No, XSLT 1.0 is enough for me

...

Take your chance to influence Microsoft's decision on XSLT 2.0 and win XSLT 2.0 book!

My advice to Oleg, if you want to see XSLT 2.0 in the .NET Framework then gather some like minded souls and build it yourself. Efforts like the MVP.XML library for the .NET Framework shows that there are a bunch of talented developers building cool enhancements to the basic XML story Microsoft provides in the .NET Framework.

I'm not sure how an informal survey in a blog would convince Microsoft one way or the other about implementing a technology. A business case to convince a product team to do something usually involves showing them that they will lose or gain significant marketshare or revenue by making a technology choice. A handful of XML geeks who want to see the latest and greatest XML specs implemented by Microsoft does not a business case make. Unfortunately, this means that Microsoft will tend to be a follower and not a leader in such cases because customer demand and competitive pressure don't occur until other people have implemented and are using the technology. Thus if you want Microsoft to implement XSLT 2.0, you're best bet is to actually have people using it on other platforms or on Microsoft platforms who will clamor for better support instead of relying on informal surveys and comments in your blog.

Just my $0.02 as someone who used to work on the XML team at Microsoft.

 

Tim Bray has a post entitled Thought Experiments where he writes

To keep things short, let’s call OpenDocument Format 1.0 "ODF" and the Office 12 XML File Formats "O12X".

Alternatives · In ODF we have a format that’s already a stable OASIS standard and has multiple shipping implementations. In O12X we have a format that will become a stable ECMA standard with one shipping implementation sometime a year or two from now, depending on software-development and standards-process timetables. ODF is in the process of working its way through ISO, and O12X will apparently be sent down that road too, which should put ISO in an interesting situation.

On the technology side, the two formats are really more alike than they are different. But, there are differences: O12X's design center, Microsoft has said repeatedly, is capturing the exact semantics of the billions of existing Microsoft Office documents. ODF’s design center is general-purpose reusability, and leveraging existing standards like SVG and MathML and so on.

Which do you like better? I know which one I’d pick. But I think we’re missing the point.

Why Are There Two? · Almost all office documents are just paragraphs of text, with some bold and some italics and some lists and some tables and some pictures. Almost all spreadsheets are numbers and labels, with some sums and averages and pivots and simple algebra. Almost all presentations are lists of bullet points with occasional pictures.

The capabilities of ODF and O12X are essentially identical for all this basic stuff. So why in the flaming hell does the world need two incompatible formats to express it? The answer, obviously, is, "it doesn’t".

I find it extremely ironic that one of the driving forces behind creating a redundant and duplicative XML format for website syndication would be one of the first to claim that we only need one XML format to solve any problem. For those who aren't in the know, Tim Bray is one of the chairs of the Atom Working Group in the IETF whose primary goal is to create a competing format to RSS 2.0 which does basically the same thing. In fact Tim Bray has written a decent number of posts attempting to explain why we need multiple XML formats for syndicating blog posts, news and enclosures on the Web.

But let's ignore the messenger and focus on the message. Tim Bray's question is quite fair and in fact he answers it later on in his blog entry. As Tim Bray writes, "Microsoft wants there to be an office-document XML format that covers their billions of legacy documents". That's it in a nutshell. Microsoft created XML versions of its binary document formats like .doc and .xls that had full fidelity with the features of these formats. That way a user can convert a 'legacy' binary Office document to a more interoperable Office XML document without worrying about losing data, formatting or embedded rich media. This is a very important goal for the Microsoft Office team and very different from the goal of the designers of the OpenDocument format. 

Is it technically possible to create a 'common shared office-XML dialect for the basics' as Tim Bray suggests? It is. It'll probably take several years (e.g. the Atom syndication format which is simply a derivative of RSS has taken over two years to come to fruition) and once it is done, Microsoft will have to 'embrace and extend' it to meet its primary goal of 100% backwards compatibility with its legacy formats. And that doesn't answer the question of what Microsoft should ship in the meantime with regards to file formats in its Office products. After all, Office 12 is scheduled to ship in the second half of 2006.

There is no simple technical solution on the horizon that will change the fact that there are be multiple XML formats for Office documents. What we need to agree on is the best way forward, not attempt to demonize each other for trying to do what's best for our customers.

Disclaimer: I work at Microsoft. However I do not work in any area related to the Office XML formats. The above is my personal opinion and should not be construed as an expression of the opinions, intents or strategies of my employer.

 

Last week Andrew Conrad told me to check out a recent article by Adam Bosworth in the ACM Queue because he wondered what I thought about. I was rather embarassed to note that althought I'd seen some mention of it online, I hadn't read it. I read it today and as usual, Adam Bosworth is on point.

The article is entitled Learning from THE WEB and it begins by listing eight "unintuitive lessons" we have learned from the Web. The lessons are listed below

1. Simple, relaxed, sloppily extensible text formats and protocols often work better than complex and efficient binary ones.

2. It is worth making things simple enough that one can harness Moore’s law in parallel.

3. It is acceptable to be stale much of the time.

4. The wisdom of crowds works amazingly well.

5. People understand a graph composed of tree-like documents (HTML) related by links (URLs).

6. Pay attention to physics.

7. Be as loosely coupled as possible.

8. KISS. Keep it (the design) simple and stupid.

Where the paper gets interesting is that then tries to apply these lessons to XML. Remember that Adam was one of the founder of the XML team at Microsoft and knows a thing or two about it. So he writes

In my humble opinion, however, we ignored or forgot lessons 3, 4, and 5. Lesson 3 tells us that elements in XML with values that are unlikely to change for some known period of time (or where it is acceptable that they are stale for that period of time, such as the title of a book) should be marked to say this. XML has no such model.
...
Lesson 4 says that we shouldn’t over-invest in making schemas universally understood.
...
Lessons 1 and 5 tell us that XML should be easy to understand without schemas

I totally agree with his assessment of the lessons learned from lessons 4 & 5. However the issue of being able to mark an element in an XML file as 'relatively unchanging' in a generic way seems to be lost on me. He then goes on to point out more of the problems with XML [and the Semantic Web/RDF]

There are some interesting implications in all of this.

One is that the Semantic Web is in for a lot of heartbreak. It has been trying for five years to convince the world to use it. It actually has a point. XML is supposed to be self-describing so that loosely coupled works. If you require a shared secret on both sides, then I’d argue the system isn’t loosely coupled, even if the only shared secret is a schema. What’s more, XML itself has three serious weaknesses in this regard:

1. It doesn’t handle binary data well.
2. It doesn’t handle links.
3. XML documents tend to be monolithic.

Now it's gotten pretty interesting and at this point, Adam throws the curve ball.

Recently, an opportunity has arisen to transcend these limitations. RSS 2.0 has become an extremely popular format on the Web. RSS 2.0 and Atom (which is essentially isomorphic) both support a base schema that provides a model for sets. Atom’s general model is a container (a <feed>) of <entry> elements in which each <entry> may contain any namespace scoped elements it chooses (thus any XML), must contain a small number of required elements (<id>, <updated>, and <title>), and may contain some other well-known ones in the Atom namespace such as <link>s. Even better, Atom clearly says that the order doesn’t matter.This immediately gives a simple model for sets missing in XML.
...
Atom also supports links of other sorts, such as comments, so clearly an Atom entry can contain links to related feeds (e.g., Reviews for a Restaurant or Complaints for a Customer) or links to specific posts. This gives us the network and graph model that is missing in XML. Atom contains a simple HTTP-based way to INSERT, DELETE, and REPLACE s within a . There is a killer app for all these documents because the browsers already can view RSS 2.0 and Atom and, hopefully, will soon natively support the Atom protocol as well, which would mean read and write capabilities.

Now that's deep. Why not move up one level of abstraction from exchanging XML documents to exchanging Web Feeds (RSS/Atom documents)? Adam ends his article by throwing a challenge out to database vendors who he believes have failed to learn the lessons of the Web by writing

All of this has profound implications for databases. Today databases violate essentially every lesson we have learned from the Web.

1. Are simple relaxed text formats and protocols supported? No.
2. Have databases enabled people to harness Moore’s law in parallel? This would mean that databases could scale more or less linearly to handle both the volume of the requests coming in and even the complexity. The answer is no.
3. Do databases optimize caching when it is OK to be stale? No.
4. Do databases let schemas evolve for a set of items using a bottom-up consensus/tipping point? Obviously not.
5. Do databases handle flexible graphs (or trees) well? No, they do not.
6. Have the databases learned from the Web and made their queries simple and flexible? No, just ask a database if it has anyone who, if they have an age, are older than 40; and if they have a city, live in New York; and if they have an income, earn more than $100,000. This is a nightmare because of all the tests for NULL.

The article ends by arguing that database vendors should add native support for the Atom Protocol and wire format. I find this interesting since based on conversations on the atom-protocol list, it is clear that Google is very interested in the Atom API. Perhaps they have already built this Atom store that Adam is arguing for and will expose the Atom API as a way to interact with it. Perhaps this Atom store accessible via Atom feeds and the Atom API is Google Base? Speculation is fun.

As for me, I tend to agree with Adam that moving up layers of abstraction is a good idea. We've all agreed on XML, the next thing to do is to agree on applications of XML. We've all agreed on RSS, the next thing to do is figure out what scenarios are enabled by the subscribe model. This is one of the reasons why I disliked the unnecessary fragmentation caused by the RSS vs. Atom battles. As for whether we need to start seeing databases with native RSS/Atom support, I think it's too early in the game to jump there. Heck, RDF has been around for a while but we are just know seeing some decent things happening with SPARQL and various RDF stores. Similarly with XML and XQuery. I don't think enough lessons have been learned from either to start thinking about what it would mean to have a native RSS/Atom store. It is an interesting idea though. 

 

A recent comment on the Groklaw blog entitled Which Binary Key? claims that one needs a "binary key" to consume XML produced by Microsoft Office 2003. Specifically the post claims

No_Axe speaks as if MS Office 12 had already been released and everyone was using it. He assumes everyone knows the binary key is gone. Yet Microsoft is saying that MS Office 12 is more or less a year away from release. So who really knows when and if the binary key has been dropped? All i know is that MSXML 12 is not available today. And that MSXML 2003 has a binary key in the header of every file.
...
So let me close with this last comment on the fabled “binary key”. In March of 2005, when phase II of the ODF TC work was complete, and the specification had been prepared for both OASIS and ISO ratification, the ODF TC took up the issue of “compliance and conformance” testing. Specifically, we decided to start work on a compliance testing suite that would be useful for developers and application providers to perfect their implementations of ODF. Guess who's XML file format was the first test target? Right. And guess what the problem is with MSXML? Right. It's the binary key. We can't do even a simple transformation between MSXML and ODF!

As someone who's used the XML features of Excel and Word, I know for a fact that you don't need a "binary key" to process the files using traditional XML tools. Brian Jones, who works on a number of the XML features in Office, has a post entitled The myth of the Binary Key where he mentions various parts of the Office XML formats that may confuse one into thinking they are some sort of "binary key" such as namespace URIs, processing instructions and Base64 encoded binary data. All of these are standard aspects of XML which one may typically doesn't see in simple uses of the technology such as in RSS feeds.

Being that I used to work on the XML team there is one thing I want to add the Brian's list which often confuses people trying to process XML; the unicode byte order mark (BOM). This is often at the beginning of documents saved in UTF-16 or UTF-8 encoding on Windows. However as the Wikipedia entry on BOM's states

In UTF-16, a BOM is expressed as the two-byte sequence FE FF at the beginning of the encoded string, to indicate that the encoded characters that follow it use big-endian byte order; or it is expressed as the byte sequence FF FE to indicate little-endian order.

Whilst UTF-8 does not have byte order issues, a BOM encoded in UTF-8 may be used to mark text as UTF-8. Quite a lot of Windows software (including Windows Notepad) adds one to UTF-8 files. However in Unix-like systems (which make heavy use of text files for configuration) this practice is not recommended, as it will interfere with correct processing of important codes such as the hash-bang at the start of an interpreted script. It may also interfere with source for programming languages that don't recognise it. For example, gcc reports stray characters at the beginning of a source file, and in PHP, if output buffering is disabled, it has the subtle effect of causing the page to start being sent to the browser, preventing custom headers from being specified by the PHP script. The UTF-8 representation of the BOM is the byte sequence EF BB BF, which appears as the ISO-8859-1 characters "" in most text editors and web browsers not prepared to handle UTF-8.

I wouldn't be surprised if the alleged "binary key" was just a byte order mark which caused problems when trying to process the XML file using non-Unicode savvy tools. I suspect some of the ODF folks who had problems with the XML file would get some use out of Sam Ruby's Just Use XML talk at this year's XML 2005 conference. 

 

There have been a number of amusing discussions in the recent back and forth between Robert Scoble and several others on whether OPML is a crappy XML format. In posts such as OPML "crappy" Robertson says and More on crappy formats Robert defends OPML. I've seen some really poor arguments made as people rushed to bash Dave Winer and OPML but  none made me want to join the discussion until this morning.

In the post Some one has to say it again… brainwagon writes

Take for example Mark Pilgrim's comments:

I just tested the 59 RSS feeds I subscribe to in my news aggregator; 5 were not well-formed XML. 2 of these were due to unescaped ampersands; 2 were illegal high-bit characters; and then there's The Register (RSS), which publishes a feed with such a wide variety of problems that it's typically well-formed only two days each month. (I actually tracked it for a month once to test this. 28 days off; 2 days on.) I also just tested the 100 most recently updated RSS feeds listed on blo.gs (a weblog tracking site); 14 were not well-formed XML.

The reason just isn't that programmers are lazy (we are, but we also like stuff to work). The fact is that the specification itself is ambiguous and weak enough that nobody really knows what it means. As a result, there are all sorts of flavors of RSS out there, and parsing them is a big hassle.

The promise of XML was that you could ignore the format and manipulate data using standard off-the-shelf-tools. But that promise is largely negated by the ambiguity in the specification, which results in ill-formed RSS feeds, which cannot be parsed by standard XML feeds. Since Dave Winer himself managed to get it wrong as late as the date of the above article (probably due to an error that I myself have done, cutting and pasting unsafe text into Wordpress) we really can't say that it's because people don't understand the specification unless we are willing to state that Dave himself doesn't understand the specification.

As someone who has (i) written a moderately popular RSS reader and (ii) worked on the XML team at Microsoft for three years, I know a thing or two about XML-related specifications. Blaming malformed XML in RSS feeds on the RSS specification is silly. That's like blaming the large number of HTML pages that don't validate on the W3C's HTML specification instead of on the fact that instead of erroring on invalid web pages web browsers actually try to render them. If web browsers didn't render invalid web pages then they wouldn't exist on the Web.

Similarly, if every aggregator rejected invalid feeds then they wouldn't exist. However, just like in the browser wars, aggregator authors consider it a competitive advantage to be able to handle malformed feeds. This has nothing to do with the quality of the RSS specification [or the HTML specification], this is all about applications trying to get marketshare.  

As for whether OPML is a crappy spec? I've had to read a lot of technology specifications in my day from W3C recommendations and IETF RFCs to API documentation and informal specs. They all suck in their own ways. However experience has thought me that the bigger the spec, the more it sucks. Given that, I'd rather have a short, human readable spec that sucks a little (e.g. RSS, XML-RPC, OPML etc.) than a large, jargon filled specificaton which sucks a whole lot more (e.g. WSDL, XML Schema, C++, etc). Then there's the issue of using the right tool for the job but I'll leave that rant for another day.

 

I've been a long time skeptic when it comes to RDF and the Semantic Web. Every once in a while I wonder if perhaps what I have a problem with is the W3C's vision of the Semantic Web as opposed to RDF itself. However in previous attempts to explore RDF I've been surprised to find that its proponents seem to ignore some of the real world problems facing developers when trying to use RDF as a basis for information integration.

Recently I've come across blog posts by RDF proponents who've begun to question the technology. The first is the blog post entitled Crises by Ian Davis where he wrote

We were discussing the progress of the Dublin Core RDF task force and there were a number of agenda items under discussion. We didn’t get past the first item though - it was so hairy and ugly that no-one could agree on the right approach. The essence of the problem is best illustrated by the dc:creator term. The current definition says An entity primarily responsible for making the content of the resource. The associated comments states Typically, the name of a Creator should be used to indicate the entity and this is exactly the most common usage. Most people, most of the time use a person’s name as the value of this term. That’s the natural mode if you write it in an HTML meta tag and it’s the way tens or hundreds of thousands of records have been written over the past six years...Of course, us RDFers, with our penchant for precision and accuracy take issue with the notion of using a string to denote an “entity”. Is it an entity or the name of an entity. Most of us prefer to add some structure to dc:creator, perhaps using a foaf:Person as the value. It lets us make more assertions about the creator entity.

The problem, if it isn’t immediately obvious, is that in RDF and RDFS it’s impossible to specify that a property can have a literal value but not a resource or vice versa. When I ask “what is the email address of the creator of this resource?” what should the (non-OWL) query engine return when the value of creator is a literal? It isn’t a new issue, and is discussed in-depth on the FOAF wiki.

There are several proposals for dealing with this. The one that seemed to get the most support was to recommend the latter approach and make the first illegal. That means making hundreds of thousands of documents invalid. A second approach was to endorse current practice and change the semantics of the dc:creator term to explictly mean the name of the creator and invent a new term (e.g. creatingEntity) to represent the structured approach.
...
That’s when my crisis struck. I was sitting at the world’s foremost metadata conference in a room full of people who cared deeply about the quality of metadata and we were discussing scraping data from descriptions! Scraping metadata from Dublin Core! I had to go check the dictionary entry for oxymoron just in case that sentence was there! If professional cataloguers are having these kinds of problems with RDF then we are fucked.

It says to me that the looseness of the model is introducing far too much complexity as evidenced by the difficulties being experienced by the Dublin Core community and the W3C HTML working group. A simpler RDF could take a lot of this pain away and hit a sweet spot of simplicity versus expressivity.

Ian Davis isn't the only RDF head wondering whether there is too much complexity involved when trying to use RDF to get things done. Uche Ogbuji also has a post entitled Is RDF moving beyond the desperate hacker? And what of Microformats? where he writes

I've always taken a desperate hacker approach to RDF. I became a convert to the XML way of expressing documents right away, in 1997. As I started building systems that managed collections of XML documents I was missing a good, declarative means for binding such documents together. I came across RDF, and I was sold. I was never really a Semantic Web head. I used RDF more as a desperate hacker with problems in a fairly well-contained domain.
...
I've developed an overall impression of dismay at the latest RDF model semantics specs. I've always had a problem with Topic Maps because I think that they complicate things in search of an unnecessary level of ontological purity. Well, it seems to me that RDF has done the same thing. I get the feeling that in trying to achieve the ontological purity needed for the Semantic Web, it's starting to leave the desperate hacker behind. I used to be confident I could instruct people on almost all of RDF's core model in an hour. I'm no longer so confident, and the reality is that any technology that takes longer than that to encompass is doomed to failure on the Web. If they think that Web punters will be willing to make sense of the baroque thicket of lemmas (yes, "lemmas", mi amici docte) that now lie at the heart of RDF, or to get their heads around such bizarre concepts as assigning identity to literal values, they are sorely mistaken. Now I hear the argument that one does not need to know hedge automata to use RELAX NG, and all that, but I don't think it applies in the case of RDF. In RDF, the model semantics are the primary reason for coming to the party. I don't see it as an optional formalization. Maybe I'm wrong about that and it's the need to write a query language for RDF (hardly typical for the Web punter) that is causing me to gurgle in the muck. Assuming it were time for a desperate hacker such as me to move on (and I'm not necessarily saying that I am moving on), where would he go from here?

Uche is one of the few RDF heads whose opinions seem grounded in practicality (Joshua Allen is another) so it is definitely interesting to see him begin to question whether RDF is the right path.

I definitely think there is some merit to disconnecting RDF from the Semantic Web and seeing if it can hang on its own from that perspective. For example, XML as a Web document format is mostly dead-on-arrival but it has found a wide variety of uses as a general data interchange format instead. I've wondered if there is similar usefulness lurking within RDF once it loses its Semantic Web baggage.

 

The announcements from about Microsoft's Linq project just keep getting better and better. In his post XML, Dynamic Languages, and VB, Mike Champion writes

Thursday at PDC saw lots of details being put out about another big project our team has been working on -- the deep support for XML in Visual Basic 9...On the VB9 front, the big news is that two major features beyond and on top of LINQ will be supported in VB9:

"XML Literals" is  the ability to embed XML syntax directly into VB code. For example,

Dim ele as XElement = <Customer/>

Is translated by the compiler to

Dim ele as XElement =  new XElement("Customer")

The syntax further allows "expression holes" much like those in ASP.NET where computed values can be inserted.

"Late Bound XML" is the ability to reference XML elements and attributes directly in VB syntax rather than having to call navigation functions.  For example

Dim books as IEnumerable(Of XElement) = bib.book

Is translated by the compiler to

Dim books as IEnumerable(Of XElement) = bib.Elements("book")

 We believe that these features will make XML even more accessible to Visual Basic's core audience. Erik Meijer, a hard core languages geek who helped devise the Haskell functional programming language and the experimental XML processing languages X#, Xen, and C-Omega, now touts VB9 as his favorite.

Erik Meijer and I used to spend a lot of time talking about XML integration into popular  programming languages back when I was on the XML team. In fact, all the patent cubes on my desk are related to work we did together in this area. I'm glad to see that some of the ideas we tossed around are going to make it out to developers in the near future. This is great news.

 

You know you're a geek when it's not even 7AM but you've already spent half the morning reading a whitepaper about Microsoft's plans to integrate XML and relational query language functionality into the .NET Framework with Linq.  C# 3.0 is going to be hot.

Like it's forefather X# Xen Cω, XLinq does an amazing job of integrating XML directly into the Common Language Runtime and the C#/VB.NET programming languages. Below are some code samples to whet your appetite until I can get around to writing an article later this year

1. Creating an XML document

   XDocument contactsDoc = 
       new XDocument(
         new XDeclaration("1.0", "UTF-8", "yes"),
         new XComment("XLinq Contacts XML Example"),
         new XProcessingInstruction("MyApp", "123-44-4444"),
           new XElement("contacts",
             new XElement("contact",
               new XElement("name","Patrick Hines"),                                       
               new XElement("phone", "206-555-0144"),
               new XElement("address",
                new XElement("street1", "123 Main St"),
                new XElement("city", "Mercer Island"),
                new XElement("state", "WA"),
                new XElement("postal", "68042")
                           )
                         )
                       )
                     );

2. Creating an XML element in the "http://example.com" namespace

   XElement contacts = new XElement("{http://example.com}contacts");

3. Loading an XML element from a file

   XElement contactsFromFile = XElement.Load(@"c:\myContactList.xml");

4. Writing out an array of Person objects as an XML file

   class Person {
           public string Name;
           public string[] PhoneNumbers;
   }

   var persons = new [] { new Person
                                                               {Name="Patrick Hines",
                                  PhoneNumbers = new string[]
                                                                                       {"206-555-0144", "425-555-0145"}
                                  },
                          new Person {Name="Gretchen Rivas",
                                      PhoneNumbers = new string[]
                                                                                       {"206-555-0163"}
                                  }
                         };

   XElement contacts = new XElement("contacts",
                           from p in persons
                           select new XElement("contact",
                               new XElement("name", p.Name),
                               from ph in p.PhoneNumbers
                               select new XElement("phone", ph)
                           )
                       );

   Console.WriteLine(contacts);

5. Print out all the element nodes that are children of the <contact> element

   foreach (x in contact.Elements()) {
               Console.WriteLine(x);
   }

6. Print all the <phone> elements that are children of the <contact> element

   foreach (x in contact.Elements("phone")) {
               Console.WriteLine(x);
   }

7. Adding a <phone> element as a child of the <contact> element

   XElement mobilePhone = new XElement("phone", "206-555-0168");
   contact.Add(mobilePhone);

8. Adding a <phone> element as a sibling of another <phone> element

   XElement mobilePhone = new XElement("phone", "206-555-0168");
   XElement firstPhone = contact.Element("phone");
   firstPhone.AddAfterThis(mobilePhone);

9. Adding an <address> element as a child of the <contact> element

   contact.Add(new XElement("address",
                 new XElement("street", "123 Main St"),
                 new XElement("city", "Mercer Island"),
                 new XElement("state", "WA"),
                 new XElement("country", "USA"),
                 new XElement("postalCode", "68042")
               ));

10. Deleting all <phone> elements under a <contact> element

    contact.Elements("phone").Remove();

11. Delete all children of the <address> element which is a child of the <contact> element

    contacts.Element("contact").Element("address").RemoveContent();

12. Replacing the content of the <phone> element under a <contact> element

    contact.Element("phone").ReplaceContent("425-555-0155");

13. Alternate technique for replacing the content of the <phone> element under a <contact> element

    contact.SetElement("phone", "425-555-0155");

14. Creating a contact element with attributes multiple phone number types

    XElement contact =
          new XElement("contact",
                new XElement("name", "Patrick Hines"),
                new XElement("phone",
                      new XAttribute("type", "home"),
                      "206-555-0144"
                ),
                new XElement("phone",
                      new XAttribute("type", "work"),
                      "425-555-0145"
                )
          );

15. Printing the value of the <phone> element whose type attribute has the value "home"

    foreach (p in contact.Elements("phone")) {
                if ((string)p.Attribute("type") == "home")
                    Console.Write("Home phone is: " + (string)p);
       }

16. Deleting the type attribute of the first <phone> element under the <contact> element

    contact.Elements("phone").First().Attribute("type").Remove();

17. Transforming our original <contacts> element to a new <contacts> element containing a list of <contact> elements whose children are <name> and <phoneNumbers>

    new XElement("contacts",
          from c in contacts.Elements("contact")
          select new XElement("contact",
                c.Element("name"),
                new XElement("phoneNumbers", c.Elements("phone"))
          )
    );

18. Retrieving the names of all the contacts from Washington, sorted alphabetically 

    from    c in contacts.Elements("contact")
    where   (string) c.Element("address").Element("state") == "WA"
    orderby (string) c.Element("name")
    select  (string) c.Element("name");

All examples were taken from the XLinq: .NET Language Integrated Query for XML Data  white paper.

 

The former co-workers (the Microsoft XML team) have been hard at work with the C# language team to bring the XML query integration into the core languages for the .NET Framework. From Dave Remy's post Anders unveils LINQ! (and XLinq) we learn

In Jim Allchin's keynote At PDC2005 today Anders Hejlsberg showed the LINQ project for the first time.  LINQ stands for Language Integrated Query.  The big idea behind LINQ is to provide a consistent query experience across different "LINQ enabled" data access technologies AND to allow querying these different data access technologies in a single query.  Out of the box there are three LINQ enabled data access technologies that are being shown at PDC.  The first is any in-memory .NET collection that you foreach over (any .NET collection that implements IEnumerable<T>).  The second is DLinq which provides LINQ over a strongly typed relational database layer.  The third, which I have been working on for the last 6 months or so (along with Anders and others on the WebData XML team), is XLinq, a new in-memory XML programming API that is Language Integerated Query enabled.  It is great to get the chance to get this technology to the next stage of development and get all of you involved.  The LINQ Preview bits (incuding XLinq and DLinq) are being made available to PDC attendees.  More information on the LINQ project (including  the preview bits) are also available online at http://msdn.microsoft.com/netframework/future/linq

This is pretty innovative stuff and I definitely can't wait to download the bits when I get some free time. Perhaps I need to write an article exploring LINQ for XML.com the way I did with my Introducing C-Omega article? Then again, I still haven't updated my C# vs. Java comparison to account for C# 2.0 and Java 1.5. It looks like I'll be writing a bunch of programming language articles this fall. 

Which article would you rather see?

 

In response to my post Using XML on the Web is Evil, Since When? Tantek updated his post Avoiding Plain XML and Presentational Markup. Since I'm the kind of person who can't avoid a good debate even when I'm on vacation I've decided to post a response to Tantek's response. Tantek wrote

The sad thing is that while namespaces theoretically addressed one of the problems I pointed out (calling different things by the same name), it actually WORSENED the other problem: calling the same thing by different names. XML Namespaces encouraged document/data silos, with little or no reuse, probably because every person/political body defining their elements wanted "control" over the definition of any particular thing in their documents. The <svg:a> tag is the perfect example of needless duplication.

And if something was theoretically supposed to have solved something but effectively hasn't 6-7 years later, then in our internet-time-frame, it has failed.

This is a valid problem in the real world. For example, for all intents an purposes an <atom:entry> element in an Atom feed is semantically equivalent to an <item> element in an RSS feed to every feed reader that supports both. However we have two names for what is effectively the same thing as far as an aggregator developer or end user is concerned.

The XML solution to this problem has been that it is OK to have myriad formats as long as we have technologies for performing syntactic translations between XML vocabularies such as XSLT. The RDF solution is for us to agree on the semantics of the data in the format (i.e. a canonical data model for that problem space) in which case alternative syntaxes are fine and we performs translations using RDF-based mapping technologies like DAML+OIL or OWL. The microformat solution which Tantek espouses is that we all agree on a canonical data model and a canonical syntax (typically some subset of [X]HTML).

So far the approach that has gotten the most traction in the real world is XML. From my perspective, the reason for this is obvious; it doesn't require that everyone has to agree on a single data model or a single format for that problem space.  

Microformats don't solve the problem of different entities coming up with the different names for the same concept. Instead its proponents are ignoring the reasons why the problem exists in the first place and then offering microformats as a panacea when they are not.

I personally haven't seen a good explanation of why <strong> is better than <b>...

A statement like that begs some homework. The accessibility, media independence, alternative devices, and web design communities have all figured this out years ago. This is Semantic (X)HTML 101. Please read any modern web design book like those on my SXSW Required Reading List, and we'll continue the discussion afterwards.

I can see the reasons for a number of the semantic markup guidelines in the case of HTML. What I don't agree with is jumping to the conclusion that markup languages should never have presentational markup. This is basically arguing that every markup language that may be used as a presentation format should use CSS or invent a CSS equivalent. I think that is a stretch.

Finally, one has to seriously cast doubt on XML opinions on a page that is INVALID markup. I suppose following the XML-way, I should have simply stopped reading Dare's post as soon as I ran into the first well-formedness error. Only 1/2 ;)

The original permalink to Tantek's article was broken after he made teh edit. I guess since I couldn't find it, it doesn't exist. ;)

 

I've been reading some of the hype around microformats in certain blogs with some amusement. I have been ignoring microformats but now I see that some of its proponents have started claiming that using XML on the Web is bad and instead HTML is the only markup language we'll ever need.

In her post Why generic XML on the Web is a bad idea Anne van Kesteren writes

Of course, using XML or even RDF serialized as XML you can describe your content much better and in far more detail, but there is no search engine out there that will understand you. For RDF there is a chance one day they will. Generic XML on the other hand will always fail to work. (Semantics will not be extracted.)

An example that shows the difference more clearly:

<em>Look at me when I talk to you!</em>

… and:

<angry>Look at me when I talk to you!</angry>

The latter element describes the content probably more accurately, but on ‘the web’ it means close to nothing. Because on the web it’s not humans who come by and try to parse the text, they already know how to read something correctly. No, software comes along and tries to make something meaningful of the above. As the latter is in a namespace no software will know and the latter is also not specified somewhere in a specification it will be ignored. The former however has been here since the beginning of HTML — even before it’s often wrongly considered presentational equivalent I — and will be recognized by software.

This post in itself isn't that bad, if anything it is just somewhat misguided. However Tantek Celik followed it up with his post Avoiding plain XML and presentational markup which boggled my mind. Tantek wrote

The marketing message of XML has been for people to develop their own tags to express whatever they wanted, rather than being stuck with the limited predefined tag set in HTML. This approach has often been labeled "plain XML" or "generic XML" or "SGML, but easier, better, and designed just for the Web".

The problem with this approach is that while having the freedom to make up all your own tags and attributes sounds like a huge improvement over the (mostly perceived) limits of HTML, making up your own XML has numerous problems, both for the author, and for users / readers, especially when sharing with others (e.g. anything you publish on the Web) is important.

This post by no means contains a complete set of arguments against plain/generic XML and presentational markup, nor are the arguments presented as definitive proofs. Mostly I wanted to share a bunch of reinforcing resources in one place. Readers are encouraged to improve upon the arguments made here.

The original impetus for creating XML was to enable SGML on the Web. People had become frustrated with the limited tag set in HTML and the solution was to create a language that enabled content creators to create their own tags yet have them still readable in browsers via stylesheet technologies (e.g. CSS). Over time, XML has failed to take off as a generic document format used by content authors for creating human readable documents on the Web but has become popular as a data format used for machine to machine communications on the Web(RSS, XML-RPC, SOAP, etc) .

Thus any arguments against XML usage on the Web today are really arguing about using XML as a data format since it isn't really used as a document format except for XHTML [and even that is only by markup geeks like Tantek & Anne].

Anyway let's look at some of Tantek's arguments against using XML on the Web...

Tower of Babel Problem

If everyone invents their own tags and attributes, pretty soon you get people calling the same thing by different names and different things by the same name. While avoid both of those occurences completely is very difficult (many of the microformats principles are design to help avoid those problems), downright encouraging authors to make up their own tags and attributes makes it much worse and all you end up with are a bunch of documents that give you the illusion of self-description.

Didn't the XML world solve this with XML namespaces like six or seven years ago?

Temptation of Presentational Markup

What happens all too often when authors or developers make up their own tags is that they choose tags that are tightly tied to a specific presentation rather than abstracting them with semantics. Quite similar to the phenomenon of authors picking presentational class names.

As a casual user of HTML, I personally haven't seen a good explanation of why <strong> is better than <b> so arguments whose entire basis is "presentational markup is evil" don't carry much weight in my book. If I come up with a custom markup format and it has a <bold> element, is that really so evil? I'm pretty sure that the XML formats used by OpenOffice or Microsoft Office contain markup that is presentational in nature whether it is setting font sizes, text colors or paragraph alignemnt. Are they evil or does the fact that they aren't intended for the Web give them a pass?

Preferring Semantic Richness

Sometimes something is a bad idea not just in absolute terms, but also relative to other approaches and solutions.

A while ago I wrote about a semantic richness spectrum on the www-style mailing list which went into a bit more detail. Håkon Wium Lie wrote a paper that both predated my rough summary by a couple of years, and provided a much more thorough analysis.

 Languages with well-known semantics are preferred to proprietary/made-up XML. This is for many reasons, including accessibility, cross-device support, and future user agent support.

This seems to be arguing that instead of cooking up your own custom format you should pick an established format with the semantics you want if one exists. This is regularly practiced in the XML world especially when it comes to the Web so I don't see how this is an argument against using XML.

--

Seriously, I feel like I am in some bizarre alternate universe if having aggregators subscribe to HTML web pages is being advocated as being a better idea than using specialized XML formats like RSS & Atom.

That's it...I'm going back to my vacation. The world has gone too loopy for me.

 

I stumbled on Bus Monster last week and even though I don't take the bus I thought it was a pretty cool application. There's a mapping application that I've been wanting for a few years and I instantly realized that given the Google Maps API I could just write it myself.

Before starting I shot a mail off to Chandu and Steve on the MSN Virtual Earth team and asked if their API would be able to support building the application I wanted. They were like "Hell Yeah" and instead of working on my review I started hacking on Virtual Earth. In an afternoon hacking session, I discovered that I could build the app I wanted and learned new words like geocoding.

My hack should be running internally on my web server at Microsoft before the end of the week. Whenever Virtual Earth goes live I'll move the app to my personal web site. I definitely learned something new with this application and will consider Hacking MSN Virtual Earth as a possible topic for a future Extreme XML column on MSDN. Would anyone be interested in that?

 

Today I learned that Apple brings podcasts into iTunes which is excellent news. This will definitely push subscribing to music and videos via RSS feeds into the mainstream. I wonder how long it'll take MTV to start providing podcast feeds.

One interesting aspect of the announcement which I didn't see in any of the mainstream media coverage was pointed out to me in Danny Ayers's post Apple - iTunes - Podcasting where he wrote

Apple - iTunes - Podcasting and another RSS 2.0 extension (PDF). There are about a dozen new elements (or “tags” as they quaintly describe them) but they don’t seem to add anything new. I think virtually everything here is either already covered by RSS 2.0 itself, except maybe tweaked to apply to the podcast rather than the item.
They’ve got their own little category taxonomy and this delightful thing:

<itunes :explicit>
This tag should be used to note whether or not your Podcast contains explicit material.
There are 2 possible values for this tag: Yes or No

I wondered at first glance whether this was so you could tell when you were dealing with good data or pure tag soup. However, the word has developed a new meaning:

If you populate this tag with “Yes”, a parental advisory tag will appear next to your Podcast cover art on the iTunes Music Store
This tag is applicable to both Channel & Item elements.

So, in summary it’s a bit of a proprietary thing, released as a fait accompli. Ok if you’re targetting for iTunes, for anything else use Yahoo! Media RSS . I wonder where interop went.

This sounds interesting. So now developers of RSS readers that want to consume podcasts have to know how to consume the RSS 2.0 <enclosure> element, Yahoo!'s extensions to RSS and Apple's extensions to RSS to make sure they cover all the bases. Similarly publishers of podcasts also have to figure out which ones they want to publish as well.

I guess all that's left is for Real Networks and Microsoft to publish their own extensions to RSS for dealing with providing audio and video metadata in RSS feeds to make it all complete. This definitely complicates my plans for adding podcasting support to RSS Bandit. And I thought the RSS 1.0 vs. RSS 2.0 vs. Atom discussions were exciting. Welcome to the world of syndication.

PS: The title of this post is somewhat tongue in cheek. It was inspired by Slashdot's headline over the weekend titled Microsoft To Extend RSS about Microsoft's creation of an RSS module for making syndicating lists work better in RSS. Similar headlines haven't been run about Yahoo! or Apple's extensions to RSS but that's to be expected since we're Microsoft. ;)

 

Joe Wilcox has a post that has me scratching my head today. In his post Even More on New Office File Formats, he writes

Friday's eWeek story about Microsoft XML-based formats certainly raises some questions about how open they really are. Assuming reporter Pater Galli has his facts straight, Microsoft's formats license "is incompatible with the GNU General Public License and will thus prevent many free and open-source software projects from using the formats." Earlier this month, I raised different concerns about the new formats openness.

To reiterate a point I made a few weeks ago: Microsoft's new Office formats are not XML. The company may call them "Microsoft Office Open XML Fromats," but they are XML-based, which is nowhere the same as being XML or open, as has been widely misreported by many blogsites and news outlets.

There are two points I'd like to make here. The first is that "being GPL compatible" isn't a definition of 'open' that I've ever heard anyone make. It isn't even the definition of Open Source or Free Software (as in speech). Heck, even the GNU website has a long list of Open Source licenses that are incompatible with the GPL. You'll notice that this list includes the original BSD license, the Apache license, the Zope license, and the Mozilla public license. I doubt that EWeek will be writing articles about how Apache and Mozilla are not 'open' because they aren't GPL compatible.

Secondly, it's completely unclear to me what distinction Joe Wilcox is making between being XML and being XML-based. The Microsoft Office Open XML formats are XML formats. They are stored on the hard drive as compressed XML files using standard compression techniques that are widely available on most platforms. Compressing an XML file doesn't change the fact that it is XML. Reading his linked posts doesn't provide any insight into whether this is the distinction Joe Wilcox is making or whether there is another. Anyone have any ideas about this?

 

 

About a year ago, the folks at Sun Microsystems came up with a bunch of benchmarks that showed that XML parsing in Java was much faster than on the .NET Framework. On the XML team at Microsoft we took this as a challenge to do much better in the next version of the .NET Framework. To see how much we improved, you can check out A comparison of XML performance on .NET 2.0 Beta2, .NET 1.1, and Sun Java 1.5 Platforms which is available on MSDN.

In the three test cases, Java 1.5 XML APIs are faster than those in the .NET Framework v1.1 both of which are about half as fast as the XML APIs in .NET Framework v2.0. The source code for the various tests is available so individuals can confirm the results for themselves on their own configurations. 

A lot of the improvements in XML parsing on the .NET Framework are due to the excellent work of Helena Kupkova. She is also the author of the excellent XmlBookMarkReader. Great stuff.

For the XML web services geeks there is also a similar comparison of XML Web Services Performance for .NET 2.0 Beta2, .NET 1.1, Sun JWSDP 1.5 and IBM WebSphere 6.0.

 

Since the recent announcement that the next version of Microsoft Office would move to open XML formats as the default file format in the next version, I've seen some questions raised about why the OpenOffice.org XML formats which were standardized with OASIS weren't used. This point is addressed in a comment by Jean Paoli in the article Microsoft to 'Open' Office File Formats which is excerpted below

"We have legacy here," Jean Paoli, Senior Microsoft XML Architect, told BetaNews. "It is our responsibility to our users to provide a full fidelity format. We didn't see any alternative; believe me we thought about it. Without backward compatibility we would have other problems."

"Yes this is proprietary and not defined by a standards body, but it can be used by and interoperable with others. They don't need Microsoft software to read and write. It is not an open standard but an open format," Paoli explained.

When asked why Microsoft did not use the OASIS (Organization for the Advancement of Structured Information Standards) OpenOffice.org XML file format, Paoli answered, "Sun standardized their own. We could have used a format from others and shoehorned in functionality, but our design needs to be different because we have 400 million legacy users. Moving 400 million users to XML is a complex problem."

There is also somewhat of a double standard at play here. The fact that we are Microsoft means that we will get beaten up by detractors no matter what we do. When Sun announced Java 1.5 5.0 with a feature set that looked a lot like those in C#, I don't remember anyone asking why they continued to invest in their proprietary programming language and platform instead of just using C# and the CLI which have been standardized by both ECMA and the ISO. If Microsoft had modified the OpenOffice.org XML file format so that it was 100% backwards compatible with the previous versions of Microsoft Office it is likely that same people would be yelling "embrace and extend". I'm glad the Office guys went the route they chose instead. Use the right tool for the job instead of trying to turn a screwdriver into a hammer.

It's a really powerful thing that the most popular Office productivity suite on the planet is wholeheartedly embracing open formats and XML. It's unfortunate that some want to mar this announcement with partisan slings and arrows instead of recognizing the goodness that will come from ending the era of closed binary document formats on the desktop.

 

About two and half years ago, I was hanging out with several members of the Office team as they gave the details about how Office 2003 would support XML file formats at XML 2002. Now that it's 2005, juicy information like that is now transmitted using blogs.

Brian Jones has a blog post entitled New default XML formats in the next version of Office were he reveals some of the details of XML support in the next version of Office. He writes

  Open XML Formats Overview

To summarize really quickly what’s going on, there will be new XML formats for Word, Excel, and PowerPoint in the next version of Office, and they will be the default for each. Without getting too technical, here are some basic points I think are important:

1. Open Format: These formats use XML and ZIP, and they will be fully documented. Anyone will be able to get the full specs on the formats and there will be a royalty free license for anyone that wants to work with the files.
2. Compressed: Files saved in these new XML formats are less than 50% the size of the equivalent file saved in the binary formats. This is because we take all of the XML parts that make up any given file, and then we ZIP them. We chose ZIP because it’s already widely in use today and we wanted these files to be easy to work with. (ZIP is a great container format. Of course I’m not the only one who thinks so… a number of other applications also use ZIP for their files too.)
3. Robust: Between the usage of XML, ZIP, and good documentation the files get a lot more robust. By compartmentalizing our files into multiple parts within the ZIP, it becomes a lot less likely that an entire file will be corrupted (instead of just individual parts). The files are also a lot easier to work with, so it’s less likely that people working on the files outside of Office will cause corruptions.
4. Backward compatible: There will be updates to Office 2000, XP, and 2003 that will allow those versions to read and write this new format. You don’t have to use the new version of Office to take advantage of these formats. (I think this is really cool. I was a big proponent of doing this work)
5. Binary Format support: You can still use the current binary formats with the new version of Office. In fact, people can easily change to use the binary formats as the default if that’s what they’d rather do.
6. New Extensions: The new formats will use new extensions (.docx, .pptx, .xlsx) so you can tell what format the files you are dealing with are, but to the average end user they’ll still just behave like any other Office file. Double click & it opens in the right application.

...

Whitepapers

The Microsoft Office Open XML Formats: New File Formats for "Office 12"

http://download.microsoft.com/download/c/2/9/c2935f83-1a10-4e4a-a137-c1db829637f5/Office12NewFileFormatsWP.doc

The Microsoft Office Open XML Formats: Preview for Developers

http://download.microsoft.com/download/c/2/9/c2935f83-1a10-4e4a-a137-c1db829637f5/Office12FileFormatDevPreviewWP.doc

This is totally awesome news. I remember asking, back in 2002, why Powerpoint didn't have an XML file format and the answer was that it was due to schedule constraints but it would be fixed in the next version. Not only did the Office guys keep their word but they went above and beyond.

This should make Sam Ruby happy.

 

It seems Jonathan Marsh has joined the blogosphere with his new blog Design By Committee. If you don't know Jonathan Marsh, he's been one of Microsoft's representatives at the W3C for several years and has been an editor of a variety of W3C specifications including XML:Base, XPointer Framework, and XInclude.

In his post XML Base and open content models Jonathan writes

There is a current controversy about XInclude adding xml:base attributes  whenever an inclusion is done.  If your schema doesn't allow those attributes to appear, you're document won't validate.  This surprises some people, since the invalid attributes were added by a previous step in the processing chain (in this case XInclude), rather than by hand.  As if that makes a difference to the validator!

Norm Walsh , after a false start, correctly points out this behavior was intentional.  But he doesn't go the next step to say that this behavior is vital!  The reason xml:base attributes are inserted is to keep references and links from breaking.  If the included content has a relative URI, and the xml:base attribute is omitted, the link will no longer resolve - or worse, it will resolve to the wrong thing.  Can you say "security hole"?

Sure it's inconvenient to fail validation when xml:base attributes are added, especially when there are no relative URIs in the included content (and thus the xml:base attributes are unnecessary.)  But hey, if you wanted people or processes to add attributes to your content model, you should have allowed them in the schema! 

I agree that the working group tried to address a valid concern. But this seems to me to be a case of the solution being worse than the problem. To handle a situation for which workarounds will exist in practice (i.e. document authors should use absolute URIs instead of relative URIs in documents) the XInclude working group handicapped using XInclude as part of the processing chain for documents that will be validated by XML Schema.

Since the problem they were trying to solve exists in instance documents, even if the document author don't follow a general guideline of favoring absolute URIs over relative URIs, these URIs can be expanded in a single pass using XSLT before being processed up the chain by XInclude. On the other hand if a schema doesn't allow xml:base elements everywhere (basically every XML format in existence) then one cannot use XInclude as part of the pipeline that creates the document if the final document will undergo schema validation.

I think the working group optimized for an edge case but ended up breaking a major scenario. Unfortunately this happens a lot more than it should in W3C specifications.

 

Stan Kitsis, who replaced me as the XML Schema program manager on the XML team, has a blog post about XInclude and schema validation where he writes

A lot of people are excited about XInclude and want to start using it in their projects.  However, there is an issue with using both XInclude and xsd validation at the same time.  The issue is that XInclude adds xml:* attributes to the instance documents while xsd spec forces you to explicitly declare these attributes in your schema.  Daniel Cazzulino, an XML MVP, blogged about this a few months ago: "W3C XML Schema and XInclude: impossible to use together???"

To solve this problem, we are introducing a new system.xml validation flag AllowXmlAttributes in VS2005.  This flag instructs the engine to allow xml:* attributes in the instance documents even if they are not defined in the schema.  The attributes will be validated based on their data type.

This design flaw in the aforementioned XML specifications is a showstopper that prevents one from performing schema validation using XSD on documents that were pre-processed with XInclude unless the schema designer decided up front that they want their format to be used with XInclude. This is fundamentally broken. The sad fact is that as Norm Walsh pointed out in his post XInclude, xml:base and validation this was a problem the various standards groups were aware of but decided to dump on implementers and users anyway. I'm glad the Microsoft XML team decided to take this change and fix a problem that was ignored by the W3C standards groups involved. 

 

Oleg Tkachenko has a post about one of the changes I was involved in while the program manager for XML programming models in the .NET Framework. In the post foreach and XPathNodeIterator - finally together Oleg writes

This one little improvement in System.Xml 2.0 Beta2 is sooo cool anyway: XPathNodeIterator class at last implements IEnumerable! Such unification with .NET iteration model means we can finally iterate over nodes in an XPath selection using standard foreach statement:

XmlDocument doc = new XmlDocument();
doc.Load("orders.xml");
XPathNavigator nav = doc.CreateNavigator();
foreach (XPathNavigator node in nav.Select("/orders/order"))
    Console.WriteLine(node.Value);

Compare this to what we have to write in .NET 1.X:

XmlDocument doc = new XmlDocument();
doc.Load("../../source.xml");
XPathNavigator nav = doc.CreateNavigator();
XPathNodeIterator ni = nav.Select("/orders/order");
while (ni.MoveNext())      
  Console.WriteLine(ni.Current.Value);

Needless to say - that's the case when just a dozen lines of code can radically simplify a class's usage and improve overall developer's productivity. How come this wasn't done in .NET 1.1 I have no idea.

One of the reasons we were hesitant in adding support for the IEnumerable interface to the XPathNodeIterator class is that the IEnumerator returned by the IEnumerable.GetEnumerator method has to have a Reset method. However a run of the mill XPathNodeIterator does not have a way to reset its current position. That means that code like the following has problems

XmlDocument doc = new XmlDocument();
doc.Load("orders.xml");
XPathNodeIterator it = doc.CreateNavigator().Select("/root/*");

foreach (XPathNavigator node in it) 
  Console.WriteLine(node.Name);
						
foreach (XPathNavigator node in it) 
 Console.WriteLine(node.Value); 

The problem is that after the first loop the XPathNodeIterator is positioned at the end of the sequence of nodes so the second loop should not print any values. However this violates the contract of IEnumerable and the behavior of practically every other class that implements the interface. We considered adding an abstract Reset() method to the XPathNodeIterator class in Whideby but this would have broken implementations of that class written against previous versions of the .NET Framework.

Eventually we decided that even though the implementation of IEnumerable on the XPathNodeIterator would behave incorrectly when looping over the class multiple times, this was an edge case that shouldn't prevent us from improving the usability of the class. Of course, it is probable that someone may eventually be bitten by this weird behavior but we felt the improved usability was worth the trade off.

Yes, backwards compatibility is a pain.

UPDATE: Andrew Kimball, who's one of the developers of working on XSLT and XPath technologies in System.Xml posted a comment that corrected some of my statements. It seems that some different implementation decisions were made after I left the team. He writes

"You know how I hate to contradict you, but the example you give actually does work correctly in 2.0. The implementation of IEnumerable saves a Clone of the XPathNodeIterator so that Reset() can simply reset to the saved Clone. There were a couple of limitations/problems, but neither was serious enough to forego implementing IEnumerable:

1. Performance -- 2 clones of the XPathNodeIterator must be taken, one in case Reset is called, and one to iterate over. In addition, getting the Current property must clone the current navigator so that the navigator's position is independent of the iterator's position.

2. Mid-Iteration Weirdness -- If MoveNext() is called several times on the XPathNodeIterator, and *then* GetEnumerator() is called, the enumerator will only enumerate the remaining nodes, not the ones that were skipped over. Basically, users should either use the XPathNodeIterator methods to iterate, *or* the IEnumerable/IEnumerator methods, not both."

I guess it just goes to show how quickly knowledge can get obsoleted in the technology game. :)

 

A little while ago I noticed a post by Oleg Tkachenko entitled Microsoft licensed Mvp.Xml library where he wrote

On behalf of the Mvp.Xml project team our one and the only lawyer - XML MVP Daniel Cazzulino aka kzu has signed a license for Microsoft to use and distribute the Mvp.Xml library. That effectively means Microsoft can (and actually wants to) use and distribute XInclude.NET and the rest Mvp.Xml goodies in their products. Wow, I'm glad XML MVPs could come up with something so valuable than Microsoft decided to license it.

Mvp.Xml project is developed by Microsoft MVPs in XML technologies and XML Web Services worldwide. It is aimed at supplementing .NET framework functionality available through the System.Xml namespace and related namespaces such as System.Web.Services. Mvp.Xml library version 1.0 released at January 2005 includes Common, XInclude.NET and XPointer.NET modules.

As a matter of interest - Mvp.Xml is an open-source project hosted at SourceForge.

Since Oleg wrote this I've seen other Microsoft XML MVPs mention the event including Don Demsak and Daniel Cazzulino. I think this is very cool and something I feel somewhat proud of since I helped get the XML MVP program started.

A few years ago, as part of my duties as the program manager responsible for putting together a community outreach plan for the XML team I decided that we needed an MVP category for XML. I remember the first time I brought it up with the folks who run the Microsoft MVP program and they thought it was such a weird idea since there were already categories for languages like C# and VB but XML was just a config file format and didn't need enough significant expertise. I was persistent and pointed out that a developer could be a guru at XSLT, XPath, XSD, DOM, etc without necessarily being a C# or VB expert. Eventually they buckled and an MVP category for XML was created.  Besides getting the XML Developer Center on MSDN launched, getting the XML MVP program started was probably my most significant achievement as part of my community outreach duties while on the XML team at Microsoft.

Now it is quite cool to see this community of developers contributing so much value to the .NET XML community that Microsoft has decided to license their technologies.

I definitely want to build a similar developer community around the stuff we are doing at MSN once we start shipping our APIs. I can't wait to get our next release out the door so I can start talking about stuff in more detail.

 

Daniel Steinberg has a an article entitled Bosworth's Web of Data where he discusses some of the ideas Adam Bosworth evangelized in his keynote at the MySQL Users Conference 2005. Daniel writes,

Bosworth explained that the key factors that enabled the web began with simplicity. HTTP was simple enough that any "P" language or JavaScript programmer could build applications. On the consumption side, web browsers such as Internet Explorer 4 were committed to rendering whatever they got. This meant that people could be sloppy and they didn't need to be high priests of syntax. Because it was a sloppy standard, people who otherwise couldn't have authored content did. The fact that it was a standard allowed this single, simple, sloppy, open wire format to run on every platform.
...
The challenge is to take a database and do for the web what was done for content. Bosworth explained that you "need a model that allows for massively linear scalability and federation of information that can spread effortlessly across a federated web."

Solutions that were suggested were to use XML and XQuery. The problem with XML is that unlike HTML, there is not a single grammar. This removed the simple and sloppy aspects of the web. The problem with XQuery is the time it took to finish the specification. Bosworth noted that it took more than four years and that "anything that takes four years is not worth doing. It is over-designed. Intead, take six months and learn from customers."
...
The next solution used web services, which began as an easy idea: you send an XML request and you get XML back. Instead, the collection of WS-* specs were huge and again, overly complicated. Bosworth said that this was a deliberate effort on the part of the companies that control the specs, like IBM and Microsoft, which deliberately made the specification hard, because then only they could deliver technology to do it.
...
Bosworth predicts that RSS 2.0 and Atom will be the lingua franca that will be used to consume all data from everywhere. These are simple formats that are sloppily extensible. Anyone who wants to can use these formats to consume content or to author content. Contrast this with the Semantic Web, which requires that you get a large group of people to agree on the schema of everything.

There are lots of interesting ideas here. I won't dwell on the criticisms of XQuery & WS-* mainly because I tend to agree that they are both overdesigned and complicated. I also wont dwell on the apparent contradiction inherent in claiming that the Semantic Web is doomed because it requires people to agree on the same schema for everything then proposing that everyone agree on using RSS as the schema for all data on the Web. I have a suspicion of what he sees as the difference but I'll wait for a blog post from him clarifying that.

What I find very interesting is using RSS is the data access format for the Web. RSS gained popularity as a way to syndicate blog posts and news sites but its turned out to be a lot more versatile than that. Sites like Feedster and Amazon's OpenSearch technology show you can use RSS as a mechanism for providing search results and integrating search engines respectively. Podcasting shows you can use RSS to syndicate digital media content instead of just plain old text or HTML. With Amazon's syndicated feeds one can keep abreast of when new CDs, books and more are released.

Over the weekend I wrote the MSN Spaces photo album browser page which displays slideshows of all the photos in the various albums on a particular user's MSN Spaces space. This page also can display the photos on a randomly selected space. This webpage is entirely powered by RSS. The photos are obtained from the RSS feed for the Space and the list of random spaces is obtained by querying MSN search with the query "site:spaces.msn.com photo album" and requesting the results as RSS. In fact, the information from the MSN Spaces RSS feeds is enough to build something like the Flickr related tags browser, where instead of showing related tags one could show spaces related to the user from the information in their blog roll which happens to also be provided in the RSS feed. Pretty nifty and all without requiring building a REST, SOAP or XML-RPC API.

In situations where one simply wants to expose read-only data via a service on the Web, it's looking like RSS is the technology to beat. As more and more information is exposed as RSS feeds, there will be even more interesting things people will be able to do with this technology. At Microsoft we definitely are gung ho about exposing as much data as possible via RSS and I've been amazed at how much enthusiasm there is around the opportunites in this area.   

Side Note: Yesterday while at the Microsoft Research Social Computing Symposium I was chatting with Randy Farmer, who's one of the guys behind Yahoo! 360° and Yahoo's purchase of Flickr, and I mentioned that it seemed like 2003 was the year that RSS really started to take off. This was also the year that Dave Winer froze the RSS 2.0 spec and Sam Ruby gathered all the malcontents in the XML syndication space and gave them a shiny new toy to play with in Atom. Coincidence?

 

I mentioned in a recent post that I was considering writing an article entitled Using Javascript, XMLHttpRequest and RSS to create an MSN Spaces photo album browser. It actually took less work than I thought to build a webpage that allows you to browse the photo albums in a particular person's Space or from a randomly chosen Space. 

I haven't used Javascript in about 5 years but it didn't take much to put the page together thanks mostly to the wealth of information available on the Web.

You can find the page at http://www.25hoursaday.com/spaces/photobrowser.html 

The page requires Javascript and currently only works in Internet Explorer but I'm sure that some intrepid soul could make it work in Firefox in a couple of minutes. If you can, please send me whatever changes are necessary.  

 

Every once in a while I see articles like Aaron Skonnard's Contract-First Service Development which make me shake my head in sorrow. His intentions are good but quite often advising people to design their XML Web services starting from an XSD/WSDL file instead of a more restricted model leads to more problems than what some have labelled the "code-first" approach.

For example, take this recent post to the XML-DEV mailing list entitled incompatible uses of XML Schema

I just got a call from a bespoke client (the XML guru in a large bank)
asking whether I knew of any XML Schema refactoring tools.

His problem is that one of their systems (from a big company)
does not handle recursive elements.  Another one of their
systems (from another big company) does not handle substitution
groups (or, at least, dynamic use of xsi:type.) Another of their
systems (from a third big company) does not handle wildcards.
(Some departments also used another tool that generated ambiguous
schemas.)

This is causing them a major headache: they are having to
refactor 7,000 element schemas by hand to munge them into
forms suited for each system.

Their schema-centricism has basically stuffed up the ready
interoperability they thought they were buying into with XML,
on a practical level. This is obviously a trap: moving to a
services-oriented architecture means that the providers can
say "we provide XML with a schema" and the pointy-headed bosses
can say "you service-user: this tool accepts XML with a schema
so you must use that!" and the service-user has little recourse.

This is one of the problems of contract first development that many of the consultants, vendors and pundits who are extolling its virtues fail to mention. A core fact of building XML Web services that use WSDL/XSD as the contract is that most people will use object<->XML mapping technologies to either create or consume the web services. However there are fundamental impedance mismatches between the W3C XML Schema Definition (XSD) Language and objects in a traditional object oriented programming language that ensure that these mappings will be problematic. I have written about these impedance mismatches several times over the past few years including posts such as The Impedence Mismatch between W3C XML Schema and the CLR.

Every XML Web Service toolkit that consumes WSDL/XSD and generates objects has different parts of the XSD spec that they either fail to handle or handle inadequately. Many of the folks encouraging contract first development are refusing to acknowledge that if developers  build schemas by hand for use in XML Web Services, it is likely they will end up using capabilities of XSD that are not supported by one or more of their consuming applications. The post from XML-DEV is just one example of this happening. When I was the program manager for XML Schema technologies in the .NET Framework I regularly had to help customers who had to deal with the interoperability problems they encountered because they'd read some article extolling the virtues of schema first design which failed to acknowledge the realities of the XML Web Service landscape.

From my experience "contract first" design is actually more likely to lead to interoperability problems than "code first" design. The only time this isn't the case is when the schema designer actually pays attention to use a minimal subset of XSD as opposed to using its full capabilities. This is one of the reasons I have tried to provide some guidance on what XSD features to avoid in my XML Schema Design guidelines series on XML.com.

However it is far easier to avoid these missteps if one starts from objects instead of XSD/WSDL since the expressiveness of objects is less than that of XSD which automatically means the web service contracts are less complicated. I remember getting this insight from Don Box and Doug Purdy a couple of months ago and rejecting it at the time since it seemed anti-XML but now I realize that it is actually the most practical thing to do.

 

It looks like I didn't get an Extreme XML column out last month. Work's been hectic but I think I should be able to start on a column by the end of the week and get it done before the end of the month. I have a couple of ideas I'd like to write about but as usual I'm curious as to what folks would be interested in reading about. Below are three article ideas in order of preference. 

1. Using Javascript, XMLHttpRequest and RSS to create an MSN Spaces photo album browser: The RSS feed for a space on MSN Spaces contains information about the most recent updates to a user's blog, photo album and lists. RSS items containing lists are indicated by using the msn:type element with the value "photoalbum". It is possible to build a photo album browser for various spaces by using a combination of Javascript for dynamic display and XMLHttpRequest for consuming the RSS feed. Of course, my code sample will be nowhere near as cool as the Flickr related tag browser.

2. Fun with operator overloading and XML: This would be a follow up piece to my Overview of Cω article. This article explores how one could simulate adding XML specific language extensions by overloading various operators on the System.Xml.XmlNode class.

3. Processing XML in the Real World: 10 Things To Worry About When Processing RSS feeds on the Web: This will be an attempt to distill the various things I've learned over the 2 years I've been working on RSS Bandit. It will cover things like how to properly use the System.Xml.XmlReader class for processing RSS feeds in a streaming fashion, bandwidth saving tips from GZip encoding to sending If-Modified-Since/If-None-Match headers in the request, dealing with proxy servers and authentication.

Which ones would you like to see and/or what is your order of preference?

 

My friend Derek, who's the dev lead for MSXML (the XML toolkit used by practically every Microsoft application from Office to Internet Explorer), has a blog post entitled XML use in the browser where he writes

C|Net has an article on what people have started calling AJAX. 'A'synchronous JavaScript and Xml. I have seen people using MSXML to build these kinds of web-apps for years, but only recently have people really pulled it all together enough, such as GMail or Outlook Web-Access (OWA). In fact, MSXML's much copied XMLHTTP (a.k.a. IXMLHttpRequest) (Copied by Apple and Mozilla/Firefox) was actually created basically to support the first implementation of OWA.

I've been thinking about what our customers want in future versions of MSXML. What kind of new functionality would enable easier/faster developement of new AJAX style web applications? XForms has some interesting ideas... I've been thinking about what we might add to MSXML to make it easier to develop rich DHtml applications. XForms is an interesting source of ideas, but I worry that it removes too much control. I don't think you could build GMail on XForms, for example.

The most obvious idea, would be to add some rich data-binding. Msxml already has some _very_ limited XML data-binding support. I have not looked much into how OWA or GMail work, but I bet that a significant part of the client-side jscript is code to regenerate the UI from the XML data behind the current page. Anyone who has used ASP/PHP/etc is used to the idea of some sort of loop to generate HTML from some data. What if the browser knew how to do that for you? And knew how to push back changes from editable controls? You can do that today with ADO.

Any other ideas? For those of you playing with 'AJAX' style design. What are the pain points? (Beside browser compatibility... )

If you are building applications that use XML in the browser and would like to influence the XML framework that will be used by future versions of Microsoft applications from Microsoft Office to Internet Explorer then you should go over to Derek's blog and let him know what you think.

 

Ever since the article Ajax: A New Approach to Web Applications unleashed itself on the Web I've seen the cacophony of hype surrounding Asynchronous JavaScript + XML (aka AJAX reach thunderous levels. The introduction to the essay already should make one wary about the article, it begins

Ajax isnt a technology. Its really several technologies, each flourishing in its own right, coming together in powerful new ways. Ajax incorporates:

• standards-based presentation using XHTML and CSS;
• dynamic display and interaction using the Document Object Model ;
• data interchange and manipulation using XML and XSLT ;
• asynchronous data retrieval using XMLHttpRequest ;
• and JavaScript binding everything together.

So AJAX is using Javascript and XML with the old new twist being that one communicates with a server using Microsoft's proprietary XmlHttpRequest object. AJAX joins SOA in ignominy as yet another buzzword created by renaming existing technologies which becomes a way for some vendors to sell more products without doing anything new. I agree with Ian Hixie's rant Call an apple an apple where he wrote

Several years ago, HTML was invented, and a few years later, JavaScript (then LiveScript, later officially named ECMAScript) and the DOM were invented, and later CSS. After people had been happily using those technologies for a while, people decided to call the combination of HTML, scripting and CSS by a new name: DHTML. DHTML wasn't a new technology it was just a new label for what people were already doing.

Several years ago, HTTP was invented, and the Web came to be. HTTP was designed so that it could be used for several related tasks, including:

• Obtaining a representation of a resource from a remote host using that resource's identifier (GET requests).
• Executing a procedure call on a remote host using a structured set of arguments (POST requests).
• Uploading a resource to a remote host (PUT requests).
• Deleting a resource from a remote host (DELETE requests).

People used this for many years, and then suddenly XML-RPC and SOAP were invented. XML-RPC and SOAP are complicated ways of executing remote procedure calls on remote hosts using a structured set of arguments, all performed over HTTP.

Of course you'll notice HTTP can already do that on its own, it didn't need a new language. Other people noticed this too, but instead of saying "hey everyone, HTTP already does all this, just use HTTP", they said, "hey everyone, you should use REST!". REST is just a name that was coined for the kind of architecture on which HTTP is based, and, on the Web, simply refers to using HTTP requests.

Several years ago, Microsoft invented XMLHttpRequest. People used it, along with JavaScript and XML. Google famously used it in some of their Web pages, for instance GMail. All was well, another day saved... then someone invented a new name for it: Ajax.
...
So I have a request: could people please stop making up new names for existing technologies? Just call things by their real name! If the real name is too long (the name Ajax was apparently coined because "HTTP+XML+HTML+XMLHttpRequest+JavaScript+CSS" was too long) then just mention the important bits. For example, instead of REST, just "HTTP"; instead of DHTML just "HTML and script", and instead of Ajax, "XML and script".

What I find particularly disappointing about the AJAX hype is that it has little to do with the technology and more to do with the quality of developers building apps at Google. If Google builds their next UI without the use of XML but only Javascript and HTML will we be inundiated with hype about the new JUDO approach (Javascript and Unspecified DOm methods) because it uses proprietary DOM extensions not in the W3C standard?

The software industry perplexes me. One minute people are complaining about standards compliance in various websites and browsers but the next minute Google ships websites built on proprietary Microsoft APIs and it births a new buzzword. I doubt that even the fashion industry is this fickle and inconsistent.

Postscript: I wasn't motivated to post about this topic until I saw the comments to the post Outlook Web Access should be noted as AJAX pioneer  by Robert Scoble. It seems some people felt that Outlook Web Access did not live up to the spirit of AJAX. Considering that the distinguishing characteristic of the AJAX buzzword is using XmlHttpRequest and Outlook Web Access is the reason it exists (the first version was written by the Exchange team) I find this highly disingenious. Others have pointed this out as well, such as Robert Sayre in his post Ever Wonder Why It's Called "XMLHTTPRequest"?

 

Steve Vinoski has a blog posting entitled Focus on the contract where he writes

Tim offers some extremely excellent advice (as usual) regarding what really matters when you write your services. If I may paraphrase what he says and perhaps embellish it a bit, starting from the implementation language and generating your contracts from it is just plain wrong, wrong, wrong, at least for systems of any appreciable magnitude, reach, or longevity. Instead, focusing on the contracts first is the way to go. I've written about this for many years now, starting well over a decade ago.

When you start with the code rather than the contract, you are almost certainly going to slip up and allow style or notions or idioms particular to that programming language into your service contract. You might not notice it, or you might notice it but not care. However, the guy on the other side trying to consume your service from a different implementation language for which your style or notions or idioms don't work so well will care.

Although Steve Vinoski's argument sounds convincing, there is one problem with it. It is actually much easier to make an uninteroperable Web service if one starts with the service contract instead of with object oriented code. The reason for this is quite simple and one I've harped on several times in the past; the impedance mismatch between XSD and objects is quite significant. There are several constructs in W3C XML Schema which simply have no counterpart in traditional object oriented languages which cause current XML Web service toolkits to barf when consuming them. For example, the XmlSerializer class in the .NET Framework supports about half the constructs in W3C XML Schema. Most XML Web Service toolkits support a similar number [but different set] of features of W3C XML Schema.

This isn't theoretical, more than once while I was the program manager for XML Schema technologies in the .NET Framework I had to take conference calls with customers who'd been converted to the 'contract first' religion only to find out that toolkits simply couldn't handle a lot of the constructs they were putting in their schemas.Those conversations were never easy.

The main thing people fail to realize when they go down the 'contract first' route is that it is quite likely that they have also gone down the 'XML first' route which most of them don't actually want to take. Folks like Tim Ewald don't mind the fact that sometimes going 'contract first' may mean they can't use traditional XML Web Service toolkits but instead have to resort to SAX, DOM and XSLT. However for many XML Web Service developers this is actually a problem instead of a solution.

 

David Megginson (the creator of SAX) has a post entitled The complexity of XML parsing APIs where he writes

Dare Obasanjo recently posted a message to the xml-dev mailing list as part of the ancient and venerable binary XML permathread (just a bit down the list from attributes vs. elements, DOM vs. SAX, and why use CDATA?). His message including the following:

I don’t understand this obsession with SAX and DOM. As APIs go they both suck[0,1]. Why would anyone come up with a simplified binary format then decide to cruft it up by layering a crufty XML API on it is beyond me.

[0] http://www.megginson.com/blogs/quoderat/archives/2005/01/31/sax-the-bad-the-good-and-the-controversial/

[1] http://www.artima.com/intv/dom.html

I supposed that I should rush to SAX’s defense. I can at least point to my related posting about SAX’s good points, but to be fair, I have to admit that Dare is absolutely right – building complex applications that use SAX and DOM is very difficult and usually results in messy, hard-to-maintain code.

I think this is a pivotal part of the binary XML debate. The primary argument for binary serializations of XML is that certain parties want to get the benefit of the wide array of technologies for processing XML yet retain the benefits of a binary format such as reduced size on the wire and processing time. Basically having one's cake and eating it too.

For me, the problem is that XML is already being pulled from too many directions as it is. In retrospect I realize it was foolish for me to think that the XML team could come up with a single API that would satisfy people processing business documents written in  wordprocessingML, people building distributed computing applications using SOAP or developers reading & writing to application configuration files. All of these scenarios use intersecting subsets of the full functionality of the XML specification. The SOAP specs go as far as banning some features of XML while others are simply frowned upon based on the fact that the average SOAP toolkit simply doesn't know what to do with them. One man's meat (e.g. mixed content) is another man's poison.

What has ended up happening is that we have all these XML APIs that expose a lot of cruft of XML that most developers don't need or even worse make things difficult in the common scenarios because they want to support all the functionality of XML. This is the major failing of APIs such as the .NET Framework's pull model parser class, System.Xml.XmlReader, DOM and SAX. The DOM also has issues with the fact that it tries to support conflicting data models (DOM vs. XPath) and serialization formats (XML 1.0 & XML 1.0 + XML namespaces). At the other extreme we have APIs that try to simplify XML by only supporting specific subsets of its expressivity such as the System.Data.DataSet and the System.Xml.XmlSerializer classes in the .NET Framework. The problem with these APIs is that the developer is dropped of a cliff once they reach the limits of the XML support of the API and have to either use a different API or resort to gross hacks to get what they need done. 

Unfortunately one of the problems we had to deal with when I was on the XML team was that we already had too many XML APIs as it was. Introducing more would create developer confusion but trying to change the existing ones would break backwards compatibility. Personally I'd rather see efforts being to create better toolkits and APIs for the various factions that use XML to make it easier for them to get work done than constantly churning the underlying format thus fragmenting it.

 

One of the biggest assumptions I had about software development was shattered when I started working on the XML team at Microsoft. This assumption was that standards bodies know what they are doing and produce specifications that are indisputable. However I've come to realize that the problems of design by committee affects illustrious names such as the W3C and IETF just like everyone else. These problems become even more pernicious when trying to combine technologies defined in multiple specifications to produce a coherent end to end application.

An example of the problem caused by contradictions in core specifications of the World Wide Web is summarized in Mark Pilgrim's article, XML on the Web Has Failed. The issue raised in his article is that determining the encoding to use when processing an XML document retrieved off the Web via HTTP, such as an RSS feed, is defined in at least three specifications which contradict each other somewhat; XML 1.0, HTTP 1.0/1.1 and RFC 3023. The bottom line being that most XML processors including those produced by Microsoft ignore one or more of these specifications. In fact, if applications suddenly started following all these specifications to the letter a large number of the XML documents on the Web would be considered invalid. In Mark Pilgrim's article, 40% of 5,000 RSS feeds chosen at random would be considered invalid even though they'd work in almost all RSS aggregators and be considered well-formed by most XML parsers including the System.Xml.XmlTextReader class in the .NET Framework and MSXML.

The newest example, of XML specifications that should work together but instead become a case of putting square pegs in round holes is Daniel Cazzulino's article, W3C XML Schema and XInclude: impossible to use together??? which points out

The problem stems from the fact that XInclude (as per the spec) adds the xml:base attribute to included elements to signal their origin, and the same can potentially happen with xml:lang. Now, the W3C XML Schema spec says:

3.4.4 Complex Type Definition Validation Rules

Validation Rule: Element Locally Valid (Complex Type)
...

3 For each attribute information item in the element information item's [attributes] excepting those whose [namespace name] is identical to http://www.w3.org/2001/XMLSchema-instance and whose [local name] is one of type, nil, schemaLocation or noNamespaceSchemaLocation, the appropriate case among the following must be true:

And then goes on to detailing that everything else needs to be declared explicitly in your schema, including xml:lang and xml:base, therefore :S:S:S.

So, either you modify all your schemas to that each and every element includes those attributes (either by inheriting from a base type or using an attribute group reference), or you validation is bound to fail if someone decides to include something. Note that even if you could modify all your schemas, sometimes it means you will also have to modify the semantics of it, as a simple-typed element which you may have (with the type inheriting from xs:string for example), now has to become a complex type with simple content model only to accomodate the attributes. Ouch!!! And what's worse, if you're generating your API from the schema using tools such as xsd.exe or the much better XsdCodeGen custom tool, the new API will look very different, and you may have to make substancial changes to your application code.

This is an important issue that should be solved in .NET v2, or XInclude will be condemned to poor adoption in .NET. I don't know how other platforms will solve the W3C inconsistency, but I've logged this as a bug and I'm proposing that a property is added to the XmlReaderSettings class to specify that XML Core attributes should be ignored for validation, such as XmlReaderSettings.IgnoreXmlCoreAttributes = true. Note that there are a lot of Ignore* properties already so it would be quite natural.

I believe this is a significant bug in W3C XML Schema that it requires schema authors to declare up front in their schemas where xml:lang, xml:base or xml:base may occur in their documents. Since I used to be the program manager for XML Schema technologies in the .NET Framework this issue would have fallen on my plate. I spoke to Dave Remy who toke over my old responsibilities and he's posted his opinion about the issue in his post XML Attributes and XML Schema .  Based on the discussion in the comments to his post it seems the members of my old team are torn on whether to go with a flag or try to push an errata through the W3C. My opinion is that they should do both. Pushing an errata through the W3C is a time consuming process and in the meantime using XInclude in combination with XML Schema is signficantly crippled on the .NET Framework (or on any other platform that supports both technologies). Sometimes you have to do the right thing for customers instead of being ruled by the letter of standards organizations especially when it is clear they have made a mistake.

Please vote for this bug on the MSDN Product Feedback Center. 

 

Coincidentally just as I finished reading a post by Tim Bray about Private Syndication, I got two bug reports filed almost simultaneously about RSS Bandit's support for secure RSS feeds. The first was SSL challenge for non-root certs where the user complained that instead of prompting the user when there is a problem with an SSL certificate like browsers do we simply fail. One could argue that this is the right thing to do especially when you have folks like Tim Bray suggesting that bank transactions and medical records should be flowing through RSS. However given the precedent set by web browsers we'll probably be changing our behavior. The second bug was that RSS Bandit doesn't support cookies. Many services use cookies to track authenticated users as well as provide individual views tailored to a user. Although there are a number of folks who tend to consider cookies a privacy issue, most popular websites use them and they are mostly harmless. I'll likely fix this bug in the next couple of weeks.  

These bug reports in combination with a couple more issues I've had to deal with while writing  code to process RSS feeds in RSS Bandit has givn me my inspiration for my next Extreme XML column. I suspect there's a lot of mileage that can be obtained from an article that dives deep into the various issues one deals with while processing XML on the Web (DTDs, proxies, cookies, SSL, unexpected markup, etc) which uses RSS as the concrete example. Any of the readers of my column care to comment on whether they'd like to see such an article and if so what they'd want to see covered?

 

I finished my first article since I switching jobs this weekend. It's tentatively titled Integrating XML into Popular Programming Languages: An Overview of Cω and should show up on both XML.com and my Extreme XML column on MSDN at the end of the month. I had initially planned to do the overview of Cω(C-Omega) for MSDN and do a combined article about ECMAScript for XML (E4X) & Cω for XML.com but it turned out that just an article on Cω was already fairly long. My plan is to follow up with an E4X piece in a couple of months. For the geeks in the audience who are a little curious as to exactly what the heck Cω is, here's an introduction to one of the sections of the article to whet your appetite.

The Cω Type System

The goal of the Cω type system is to bridge the gap between Relational, Object and XML data access by creating a type system that is a combination of all three data models. Instead of adding built-in XML or relation types to the C# language, the approach favored by the Cω type system has been to add certain general changes to the C# type system that make it more conducive for programming against both structured relational data and semi structured XML data.

A number of the changes to C# made in Cω make it more conducive for programming against strongly typed XML, specifically XML constrained using W3C XML Schema. Several concepts from XML and XML Schema have analogous features in Cω. Concepts such as document order, the distinction between elements and attributes, having multiple fields with the same name but different values, and content models that specify a choice of types for a given field exist in Cω. A number of these concepts are handled in traditional Object<->XML mapping technologies but it is often with awkwardness. Cω aims at makes programming against strongly typed XML as natural as programming against arrays or strings in traditional programming languages.

I got a lot of good feedback on the article from a couple of excellent reviewers including the father of the X#/Xen himself, Erik Meijer. For those not in the know, X#/Xen was merged with Polyphonic C# to create Cω. Almost all of my article focuses on the aspects of Cω inherited from X#/Xen.

 

It seems there's been some recent hubbub in the world of podcasting about how to attach multiple binary files to a single post in an RSS feed In a post entitled Multiple-enclosures on RSS items? Dave Winer weighs in on the issue. He writes

This question comes up from time to time, and I've resisted answering it directly, thinking that anyone who really read the spec would come to the conclusion that RSS allows zero or one enclosures per item, and no more. The same is true for all other sub-elements of item, except category, where multiple elements are explicitly allowed. The spec refers to "the enclosure" in the singular. Regardless, some people persist in thinking that you may have more than one enclosure per item.

Okay, let's play it out. So if I have more than one enclosure per item, how do I specify the publication date for each enclosure? How do I specify the title, author, a link to comments, a description perhaps, or a guid? The people who want multiple enclosures suggest schemes that are so complicated that they're reduced to hand-waving before they get to the spec, which I would love to read, if it could be written. Some times some things are just too hard to do. This is one of them.

And there's a reason why it's too hard. Because you're throwing out the value of RSS and then trying to figure out how to bring it back. There's no need for items any more, so you might as well get rid of them. At the top level of channel would be a series of enclosures, and then underneath each enclosure, all the meta-data. Voila, problem solved. Only what have you actually solved? You've just re-created RSS, but instead of calling the main elements "item" we now call them "enclosure".

The value of RSS is fairly self evident to me but it seems that given the amount of people who keep wanting to reinvent the wheel it may not be as clear to others. As someone who used to work on core XML technologies at Microsoft, the value of XML was obvious to me. It allowed developers to agree to use the same data format for information interchange which led to a proliferation of a wide and uniform set of tools for working with data formats. XML is not an optimal format for most of the tasks it is used for but it more than makes up for this with the plethora of tools and technologies that exist for processing XML.  

My expectation about XML was always that the software industry would move on to agreeing on other higher level protocols built on XML for application information interchange. So I've always been frustrated to see many attempts by various parties, including the W3C with efforts such as XML 1.1 and binary XML, take us steps back by wanting to fragment the interoperability promise of XML.

RSS is a wonderful example of the higher level of interoperability that can be built upon XML formats. Instead of information sources using various incompatible mechanisms for providing information to end users such as NOAA's SOAP web service and the Microsoft.com web services which each require a separate custom application to consume them, sites can all standardize on RSS. This standardization creates an ecosystem of applications that produce and consume RSS feeds which is a lot larger than what would exist for each site specific web services or market specific XML syndication formats.  Specifically, it allows for the evolution of the digital information hub where users can view data from the various information sources they care about (blogs, news, weather reports, etc) in their choice of applications. 

Additionally, RSS is extensible. This means that even if the core elements and attributes do not satisfy all the requirements of a particular problem domain, then domain-specific information can be added to the feed. This allows for regular consumers of RSS to still be able to consume the content while domain specific applications can give users a richer experience. This is a much better solution for both content producers and consumers than coming up with domain specific applications.

As a user I want less formats not more. I want my email to come in my RSS aggregator, I want my favorite newsgroups to show up in my RSS aggregator, I'm tired of having a separate application for what is essentially the same kind of data. In fact, it seems Google agrees with me as evidenced by them exposing XML feeds for your GMail inbox and for USENET newsgroups via Google Groups. Unfortunately, if you have a plain old RSS reader, you can't view these feeds and instead have to find an aggregator that supports Atom 0.3. Two steps forward, one step back.

We need less data interchange formats not more. It is better for content producers, better for end users and better for developers of applications that use these formats. Existing problems in syndication should focus on how to make the existing formats work for us instead of inventing new formats. 

Vive la RSS. 

 

Looking at the calendar I realized that I have two articles due this month, one for my Extreme XML column on MSDN and another that I promised XML.com a few months ago. I was reminded of this by the following excerpt from Ed Dumbill's recent article On Folly where he wrote

Champion cited two developments of particular interest. The first is E4X, the addition of native XML capabilities to ECMAScript. An implementation of this in the Mozilla project is currently coming to fruition. The second development is "Comega" (aka "Cω"), an extension of C# including native XML data types. (Editor's Note: Watch XML.com for a forthcoming introduction to Comega from Dare Obasanjo.)

So I'm on the hook for an overview of Cω. I started to wonder whether it wouldn't be cool if my Extreme XML column focused deeply on Cω while my XML.com article was an overview of both E4X and Cω. This would save me some effort in coming up with a separate topic for my Extreme XML column but should provide interesting information for both XMl.com readers and MSDN readers. What do you think?

 

Since I left the XML team at Microsoft they've gone on an impressive hiring spree. Two people who I'd have loved to work with who have joined up after I left are

David Remy: former Director of Product Engineering at Bea Systems, he was responsible for security, web services, and XML for Bea's Weblogic Workshop product line.

Mike Champion: formerly a research and development specialist at Software AG. He was on the W3C's Document Object Model (DOM) Working Group for over three years and was the co-chair of the Web Services Architecture Working Group.

I have to see how quickly I can nag both of them to start blogging. David used to be in charge of the folks who produced XML Beans so I suspect he has all sorts of interesting perspectives on XML programming models. Mike and I have been exchanging mail on XML-DEV for years and now that we can have some of these conversations in person, I work across campus. Irony indeed. :)

 

Tim Bray has a post entitled More Relax where he writes

I often caution people against relying too heavily on schema validation. “After all,” I say, “there is lots of obvious run-time checking that schemas can’t do, for example, verifying a part number.” It turns out I was wrong; with a little extra work, you can wire in part-number validation—or pretty well anything else—to RelaxNG. Elliotte Rusty Harold explains how. Further evidence, if any were required, that RelaxNG is the world’s best schema language, and that anyone who who’s using XML but not RelaxNG should be nervous.

Elliote Rusty Harold's article shows how to plug in custom datatype validation into Java RELAX NG validators. This enables one to enforce complex constraints on simple types such as such as "the content of an element is correctly spelled, as determined by consulting a dictionary file" or "the number is prime" to take examples from ERH's article. 

Early in the design of the version 2.0 of the System.Xml namespace in the .NET Framework we considered creating a System.Xml.IXmlType interface. This interface would basically represent the logic for plugging one's custom types into the XSD validation engine. After a couple of months and a number of interesting discussions between myself, Joshua and Andy we got rid of it.

There were two reasons we got rid of this functionality. The simple reason was that we didn't have much demand for this functionality. Whenever we had people complaining about the limitations of XSD validation it was usually due to its inability to define co-occurence constraints (i.e. if some element or attribute has a certain value then the expected content should be blah)  and other aspects of complex type validation than needing more finer grained simple type validation. The other reason was that the primary usage of XSD for many of our technologies is primarily as a type system not as a validation language. There's already the fact that XSD schemas are used to generate .NET Framework classes via the System.Xml.Serialization.XmlSerializer and relational tables via the System.Data.DataSet. However there were already impedence mismatches between these domains and XSD, for example if one defined a type as xs:nonNegativeInteger this constraint was honored in the generated C#/VB.NET classes created by the XmlSerializer or in the relational tables created by the DataSet. Then there was the additional wrinkle that at the time we were working on XQuery which used XSD as its typoe system and we had to factor in the fact that if people could add their own simple types we didn't just have to worry about validation but also how query operators would work on them. What would addition, multiplication or subtraction of such types mean? How would type promotion, casting or polymorphism work with some user's custom type defined outside the rules of XSD?

Eventually we scrapped the class as having too much cost for too little benefit.

This reminds me of Bob DuCharme's XML 2004 talk Documents vs. Data, Schemas vs. Schemas where he advised people on how to view RELAX NG and XSD. He advised viewing RELAX NG as a document validation language and considering XSD as a datatyping language. I tend to agree although I'd probably have injected something in there about using XSD + Schematron for document validation so one could get the best of both worlds.  

 

 

Adam Bosworth has posted his ISCOC04 talk on his weblog. The post is interesting although I disagreed with various bits and pieces of it. Below are some comments in response to various parts of his talk

On the one hand we have RSS 2.0 or Atom. The documents that are based on these formats are growing like a bay weed. Nobody really cares which one is used because they are largely interoperable. Both are essentially lists of links to content with interesting associated metadata. Both enable a model for capturing reputation, filtering, stand-off annotation, and so on. There was an abortive attempt to impose a rich abstract analytic formality on this community under the aegis of RDF and RSS 1.0. It failed. It failed because it was really too abstract, too formal, and altogether too hard to be useful to the shock troops just trying to get the job done. Instead RSS 2.0 and Atom have prevailed and are used these days to put together talk shows and play lists (podcasting) photo albums (Flickr), schedules for events, lists of interesting content, news, shopping specials, and so on. There is a killer app for it, Blogreaders/RSS Viewers.

Although it is clear that RSS 2.0 seems to be edging out RSS 1.0, I wouldn't say it has failed per se. I definitely wouldn't say it failed for being too formal and abstract. In my opinion it failed because it was more complex with no tangible benefit. This is the same reason XHTML has failed when compared to HTML. This doesn't necessarily mean that more rigid sysems will fail to take hold when compared to less rigid systems, if so we'd never have seen the shift from C to C++ then from C++ to C#/Java.

Secondly, it is clear It seems Adam is throwing out some Google spin here by trying to lump the nascent and currently in-progress Atom format in the same group as RSS 2.0. In fact, if not for Google jumping on the Atom bandwagon it would even be more of an intellectual curiousity than RSS 1.0.    

As I said earlier, I remember listening many years ago to someone saying contemptuously that HTML would never succeed because it was so primitive. It succeeded, of course, precisely because it was so primitive. Today, I listen to the same people at the same companies say that XML over HTTP can never succeed because it is so primitive. Only with SOAP and SCHEMA and so on can it succeed. But the real magic in XML is that it is self-describing. The RDF guys never got this because they were looking for something that has never been delivered, namely universal truth. Saying that XML couldn't succeed because the semantics weren't known is like saying that Relational Databases couldn't succeed because the semantics weren't known or Text Search cannot succeed for the same reason. But there is a germ of truth in this assertion. It was and is hard to tell anything about the XML in a universal way. It is why Infopath has had to jump through so many contorted hoops to enable easy editing. By contrast, the RSS model is easy with an almost arbitrary set of known properties for an item in a list such as the name, the description, the link, and mime type and size if it is an enclosure. As with HTML, there is just enough information to be useful. Like HTML, it can be extended when necessary, but most people do it judiciously. Thus Blogreaders and aggregators can effortlessly show the content and understanding that the value is in the information. Oh yes, there is one other difference between Blogreaders and Infopath. They are free. They understand that the value is in the content, not the device.

Lots of stuff to agree with and disagree with here. Taking it from the top, the assertion that XML is self-describing is a myth. XML is a way to attach labels to islands of data, the labels are only useful if you know what they mean. Where XML shines is that one can start with a limited set of labels that are widely understood (title, link, description) but attach data with labels that are less likely to be understood (wfw:commentRss, annotate:reference, ent:cloud) without harming the system. My recent talk at XML 2004, Designing XML Formats: Versioning vs. Extensibility, was on the importance of this and how to bring this flexibility to the straitjacketed world of XML Schema.

I also wonder who the people are that claim that XML over HTTP will never succeed. XML over HTTP already has in a lot of settings. However I'd question that it is all you need. The richer the set of interactions allowed by the web site the more an API is needed. Google, Amazon and eBay all have XML-based APIs. Every major blogging tool has an XML-based API even though those same tools are using vanilla XML over HTTP for serving RSS feeds. XML over HTTP can succeed in a lot of settings but as the richness of the interaction between client and server grows so also does the need for a more powerful infrastructure.

The issue is knowing how to pick right tool for the job. You don't need the complexity of the entire WS-* stack to build a working system. I know a number of people at Microsoft realize that this message needs to get out more which is why you've begun to see things like Don Box's WS-Why Talk and the WS Kernel.

What has been new is information overload. Email long ago became a curse. Blogreaders only exacerbate the problem. I can't even imagine the video or audio equivalent because it will be so much harder to filter through. What will be new is people coming together to rate, to review, to discuss, to analyze, and to provide 100,000 Zagat's, models of trust for information, for goods, and for services. Who gives the best buzz cut in Flushing' We see it already in eBay. We see it in the importance of the number of deals and the ratings for people selling used books on Amazon. As I said in my blog, My mother never complains that she needs a better client for Amazon. Instead, her interest is in better community tools, better book lists, easier ways to see the book lists, more trust in the reviewers, librarian discussions since she is a librarian, and so on.
This is what will be new. In fact it already is. You want to see the future. Don't look at Longhorn. Look at Slashdot. 500,000 nerds coming together everyday just to manage information overload. Look at BlogLines. What will be the big enabler' Will it be Attention.XML as Steve Gillmor and Dave Sifry hope' Or something else less formal and more organic' It doesn't matter. The currency of reputation and judgment is the answer to the tragedy of the commons and it will find a way. This is where the action will be. Learning Avalon or Swing isn't going to matter. Machine learning and inference and data mining will. For the first time since computers came along, AI is the mainstream.

I tend to agree with most of this although I'm unsure why he feels the need to knock Longhorn and Java. What he seems to be overlooking is that part of the information overload problem is the prevalance of poor data visualization and user interface metaphors for dealing with significant amounts of data. I know believe that one of the biggest mistakes I made in the initial design of RSS Bandit was modelling it after mail readers like Outlook even though I knew lots of people who had difficulty managing the flood of email they get using them. This is why the next version of RSS Bandit will borrow a leaf from FeedDemon along with some other tricks I have up my sleeve.

A lot of what I do in RSS Bandit is made easy due to the fact that it's built on the .NET Framework and not C++/MFC so I wouldn't be as quick to knock next generation GUI frameworks as Adam is. Of course, now that he works for a Web company the browser is king.

 

My XML in the .NET Framework: Past, Present & Future talk went well yesterday. The room was full and people seemed to like what they heard. The audience was most enamored with the upcoming System.Xml.Schema.XmlSchemaInference class that provides the ability to generate schemas from sample documents and the new XSLT debugger.

It was nice having people walk up to me yesterday to tell me how much they liked my talk from the previous day. There were even a couple of RSS Bandit users who walked up to me to tell me how much they liked it. This was definitely my best XML conference experience.

Arpan did comment on the irony of me giving more talks about XML after leaving the XML team at Microsoft than when I was on the team. :)

 

My XML 2004 talk, Designing XML Formats: Versioning vs. Extensibility, went over well yesterday. Lots of interesting questions were asked during the Q&A session for my talk and the following talk by Dave Orchard, Achieving Distributed Extensibility and Versioning.

One issue that came up during the discussions after our talk was the cost/benefit of using a mustUnderstand construct in an XML format similar to the SOAP mustUnderstand attribute. The primary benefit of the having such a construct is that it enables third parties to create mandatory extensions to an XML format. However there a number of costs to having such a construct

1. Entire Element or Document Must Be Read: A processor that just wants to extract a subset of the data in the document still has to parse the entire document and see if there are any mustUnderstand constructs before it can process the document. This increases the cost of processing instances of the format.
2. Ambiguity as to what is Meant by 'Understand': The concept of what it means to "understand" an XML vocabulary is context specific. For example, should a stylesheet that pretty prints an XML document fail because the format contains a mustUnderstand construct that is not explicitly handled by the stylesheet? A mustUnderstand construct is particularly limiting since it forces all consumers to fail even though there may be some consumers that can still use the format even if they don't explicitly understand certain elements or attribute in the document.
3. Causes Confusion for Intermediaries: In certain cases, a format may be processed by an intermediary on the way to the client from the server. For example, HTTP requests often pass through proxy servers and there are also web-based aggregators of RSS/Atom feeds such as Feedster & PubSub which can then be subscribed to by other aggregators. In such cases, it is ambiguous whether intermediaries are expected to fail if a construct which isn't explicitly handled is labelled as mustUnderstand or whether they are expected to pass it on with that label to third party aggregators. In fact certain formats thus have separate mustUnderstand constructs for hop-to-hop versus end-to-end transmission.

From my perspective, the cost of having a mustUnderstand construct is often not worth the benefits provided. This wasn't explicitly in my talk but is a conclusion I came to recently which I expanded upon during the Q&A session.

 

Many times when implementing XML specifications I've come across I've come up against feature that just seem infeasible or impractical to implement. However none of them have given me nightmares as they have my friend Mike Vernal, a program manager on the Indigo team at Microsoft. In his post could you stop the noise, i'm trying to get some rest ... he talks about spending nights tossing and turning having nightmares about how the SOAP mustUnderstand header attribute should be processed. In Mike's post More SOAP Sleepness he mentions having sleepless nights worrying about the behavior of SOAP intermediaries as described in Section 2.7: Relaying SOAP Messages.

This isn't to say I didn't have sleepless nights over implementing XML specifications when I worked on the XML team at Microsoft. One of the issues that consumed a lot more of my time than is reasonable is explained in Derek Denny-Brown's post Loving and Hating XML Namespaces 

Namespaces and your XML store
For example, load this document into your favorite XML store API (DOM/XmlBeans/etc)
 <book title='Loving and Hating XML Namespaces'>
   <author>Derek Denny-Brown</author>
 </book>
Then add the attribute named "xmlns" with value "http://book" to the <book> element. What should happen? Should that change the namespaces of the <book> and <author> elements? Then what happens if someone adds the element <ISBN> (with no namespace) under <book>? Should the new element automatically acquire the namespace "http://book", or should the fact that you added it with no namespace mean that it preserves it's association with the empty namespace?

In MSXML, we tried to completely disallow editing of namespace declarations, and mostly succeeded. There was one case, which I missed, and we have never been able to fix it because so many people found it an exploited it. The W3C's XML DOM spec basically says that element/attribute namespaces are assigned when the nodes are created, and never change, but is not clear about what happens when a namespace declaration is edited.

Then there is the problem of edits that introduce elements in a namespace that does not have an existing namespace declaration:
<a xmlns:p="http://p/">
  <b>
    ...
      <c p:x="foo"/>
    ...
  </b>
</a>
If you add attribute "p:z" in namespace "bar" to element <b>, what should happen to the p:x attribute on <c>? Should the implementations scan the entire content of <b> just in case there is a reference to prefix "p"?

Or what about conflicts? Add attribute "p:z" in namespace "bar" to the below sample... what should happen?
<a xmlns:p="http://p/" p:a="foo"/>

This problem really annoyed me while I was the PM for the System.Xml.XmlDocument class and the short-lived System.Xml.XPath.XPathDocument2. In the former, I found out that once you started adding, modifying and deleting namespace declarations the results would most likely be counter-intuitive and just plain wrong. Of course, the original W3C DOM spec existed before XML namespaces and trying to merge them in after the fact was probably a bad idea. With the latter class, it seemed the best we could do was try and prevent editing namespace nodes as much as possible. This is the track we decided to follow with the newly editable System.Xml.XPath.XPathNavigator class in the .NET Framework.

This isn't the most sleep depriving issue I had to deal with when trying to reflect the decisions in various XML specifications in .NET Framework APIs. Unsurprisingly, the spec that caused the most debate amongst our team when trying to figure out how to implement its features over an XML store was the W3C XML Schema Recommendation part 1: Structures. The specific area was the section on contributions to the Post Schema Validation Infoset and the specific infoset contribution which caused so much consternation was the validity property.

After schema validation an XML element or attribute should have additional metadata added to it related to validation such as it's type, its default value specified in the schema if any and whether it is valid or not according to its type. Although the validity property is trivial to implement on a read-only API such as the System.Xml.XmlReader class, it was unclear what would be the right way to expose this in other representations of XML data such as the System.Xml.XmlDocument class. The basic problem is "What happens to the validity propety of the element or attribute those of all its ancestors once the node is updated?". Once I change the value of an age element which is typed as an integer from 17 to seventeen what should happen. Should the DOM test every edit to make sure it is valid for that type then reject it otherwise? Should the edit be allowed but the validity property of the element and all its ancestors be changed? What if there is a name element with required first and last elements and the user wants to delete the first element and replace it with a different one? How would that be reflected with regards to the validity property of the name element?

None of the answers to the question we came up with satisfactory. In the end, we were stuck between a rock and a hard place so we made the compromise choice. I believe we debated this issue every other month for about a year.

 

At Chris Sells' XML DevCon conference* Don Box gave a talk called WS-Why which is described below

"Why? This talk will make sense of why various WS-* specs came to life and which ones every developer should ignore. Naturally, the size of this set is non-zero, however, it is not the entire universe. Hopefully, the audience will be left with a mental model for what to ignore going forward as the WS-* machine continues to move forward."

I got to hang out with Don before the conference as well as read the slides for his talk and although I liked the direction of the talk I wished it could have been more prescriptive. Before continuing to read it's a good idea to read a summary of Don's talk such as the one  at Jeff Barr's blog post AXDC - Don Box and WS-Why?.

In his talk Don breaks XML Web Services specs into

• WS-DesertIsland - specs that are a must have that form the core XML Web Service specs. These include XML, SOAP, WS-Addressing, WS-MetadataExchange &  XSD+WSDL
• WS-IslandResort - the next layer of important specs after the core. These include WS-Security, WS-Trust, WS-ReliableMessaging & WS-Policy
• WS-NewZealand - specs you'd probably need once in a lifetime. These include WS-Eventing, WS-Enumeration & WS-AtomicTransaction
• WS-IslandOfDoctorMoreau - the ugly step children of the WS-* spec family. These include UDDI, WS-Transfer, WS-BusinessActivity, MTOM and BPEL4WS
• WS-FantasyIsland - specs Don would love to see. These include WS-Data (XPath/SQL-like query for web services), SOAP over TCP, XSD with better support for versioning, binary XML & WSDL based on RELAX NG.

As can be expected there have been folks who've done a deeper analysis of Don's talk than what I've done above. The most significant so far has been Steve Maine's post The Web Services Kernel which gives an overview of the 5 specs in Don's WS-DesertIsland.

In general I agree with the direction Don took with the talk. However although it was an appropriate talk for the audience of the XML Devcon, a bunch of implementers and industry experts, I don't see it as significant guidance to developers trying to make sense of the mess of WS-* specs. Don's talk is best seen through the lens of "If I was an implementer which specs should I implement in my XML Web Services toolkit", not "If I was a developer which specs should I use from my XML Web Services toolkit". This is an important distinction. This is why specs like WS-Addressing are in Don's WS kernel even though it is only important if you aren't using HTTP as your transport which most developers would be. 

The talk I'd most love to see next from Don or whoever else in Indigo is going to be doing the conference route next is WS-Which: How to decide what XML Web Services specs are right for your application. As someone who now has the responsibility of designing XML Web Service end points within an intranet (aren't job switches grand?) and perhaps beyond I'm interested in guidance that explains when I should use WS-Security versus SOAP + HTTPS or whether there are any scenarios where using MTOM or WS-Transfer actually make sense.

Don's talk was a good start by the Indigo team for providing guidance for future users of the XML Web Services family of specs but there's still a lot of guidance that is currently missing from them to the industry. More importantly, just because some spec is or isn't in Don's WS kernel doesn't indicate how significant or not it will be to a particular class of application developers. Perhaps I can get Doug to give this talk next year.

* Someone really needs to explain to Chris Sells how the Web works. Constantly changing the content of the page at http://www.sellsbrothers.com/conference logically breaks links to the site

 

Karl Waclaweck has released version 1.0 of the SAX for .NET project. In the announcement on XML-DEV Karl writes

This is the first production release of the C#/.NET port of the SAX API.
It should be compatible with MS.NET 1.1 and Mono 1.0.2.

Since the API alone is not enough, a SAX parser implementation has been
released as well: SAXExpat.NET 1.0. It is based on the Expat parser, and
will work on MS.NET 1.1. Currently Mono 1.0.2 is not able to run it,
but this will hopefully change with future Mono releases.

Another implementation based on a port of the AElfred parser will
be available soon. It should work under both, MS.NET and Mono.

The project page is here: http://saxdotnet.sourceforge.net/

It's good to see more Open Source XML projects showing up for the .NET Framework. I haven't missed using SAX that much but I imagine people coming to the .NET Framework from the Java world would love to be able to keep using their favorite push model XML parsing API.

 

Derek Denny-Brown, the dev lead for both MSXML & System.Xml, who's been involved with XML before it even had a name has finally started a blog. Derek's first XML-related post is Where XML goes astray... which points out three features of XML that turn out to have caused significant problems for users and implementers of XML technologies. He writes

First, some background: XML was originally designed as an evolution of SGML, a simplification that mostly matched a lot of then existing common usage patterns. Most of its creators saw XML and evolving and expanding the role of SGML, namely text markup. XML was primarily intended to support taking a stream of text intended to be interpreted as a human readable document, and delineate portions according to some role. This sequence of characters is a paragraph. That sequence should be displayed with a link to some other information. Et cetera, et cetera. Much of the process in defining XML based on the assumption that the text in an XML document would eventually be exposed for human consumption. You can see this in the rules for what characters are allowed in XML content, what are valid characters in Names, and even in "</tagname>" being required rather than just "</>".
...
Allowed Characters
The logic went something like this: XML is all about marking up text documents, so the characters in an XML document should conform to what Unicode says are reasonable for a text document. That rules out most control characters, and means that surrogate pairs should be checked. All sounds good until you see some of the consequences. For example, most databases allow any character in a text column. What happens when you publish your database as XML? What do you do about values that include characters which are control characters that the XML specification disallowed? XML did not provide any escaping mechanism, and if you ask many XML experts they will tell you to base64 encode your data if it may include invalid characters. It gets worse.

The characters allowed in an XML name are far more limited. Basically, when designing XML, they allowed everything that Unicode (as defined then) considered a ‘letter’ or a ‘number’. Only 2 problems with that: (1) It turns out many characters common in Asian texts were left out of that category by the then-current Unicode specification. (2) The list of characters is sparse and random, making implementation slow and error prone.
...
Whitespace
When we were first coding up MSXML, whitespace was one of our perpetual nightmares. In hand-authored XML documents (the most common form of documents back then), there tended to be a great deal of whitespace. Humans have a hard time reading XML if everything is jammed on one line. We like a tag per line and indenting. All those extra characters, just there so that our feeble minds could make sense of this awkward jumble of characters, ended up contributing significantly to our memory footprint, and caused many problems to our users. Consider this example:
 <customer>  
           <name>Joe Schmoe</name>  
           <addr>123 Seattle Ave</addr> 
  </customer>
A customer coming to XML from a database back ground would normally expect that the first child of the <customer> element would be the <name> element. I can’t explain how many times I had to explain that it was actually a text node with the value newline+tab.
...
XML Namespaces
Namespaces is still, years after its release, a source of problems and disagreement. The XML Namespaces specification is simple and gets the job done with minimum fuss. The problem? It pushes an immense burden of complexity onto the APIs and XML reader/writer implementations. Supporting XML Namespaces introduces significant complexity in the parsers, because it forces parsers to parse the entire start-tag before returning any text information. It complicates XML stores, such as DOM implementations, because the XML Namespace specification only discusses parsing XML, and introduces a number of serious complications to edit scenarios. It complicates XML writers, because it introduces new constraints and ambiguities.

Then there is the issue of the 'default namespace’. I still see regular emails from people confused about why their XPath doesn’t work because of namespace issues. Namespaces is possibly the single largest obstacle for people new to XML.

My experiences as the program manager for the majority of the XML programming model in the .NET Framework agree with this list. The above list hits the 3 most common areas people seem to have problems with working with XML in the .NET Framework. His blog post makes a nice companion piece to my The XML Litmus Test: Understanding When and Why to Use XML article on MSDN.

 

We are in the process of locking down System.Xml for Beta 2 of the .NET Framework 2.0 and Visual Studio 2005. In the past few months we have received customer feedback about our feature set previewed in the Whidbey Alpha & Whidbey Beta 1 and this has guided our decision making process as to where to focus our energies to ensure a comprehensive feature set.

Below is the list of changes to System.Xml and subsidiary namespaces that have occurred between Beta 1 and Beta 2 of the .NET Framework 2.0 release.

ADDITIONS

XmlSchemaValidator

The XmlSchemaValidator class provides a push model API for W3C XML Schema validation. The primary scenario for using the XmlSchemaValidator is for validating an XML infoset in-place without having to serialize it as an XML document then reparse the document using a validating XML reader.

CHANGES

XmlReader

• Added overloads to the static Create() method that take XmlParserContext
• ReadValueAsXXX() methods renamed to ReadContentAsXXX(). Also reduced the number of ReadContentAsXXX() methods relative to the number of ReadValueAsXXX() methods in Whidbey beta 1.
• Added ReadElementContentAsXXX() methods which are specific to obtaining the value of element nodes
• Added methods for reading large streams of text or binary data embedded in an XML document in a streaming fashion

public virtual bool CanReadValueChunk { get; }

public virtual int ReadValueChunk (byte[] buffer, int startIndex, int count);

public virtual bool CanReadBinaryContent { get; }

public virtual int ReadContentAsBase64 (byte[] buffer, int startIndex, int count);

public virtual int ReadContentAsBinHex (byte[] buffer, int startIndex, int count);

public virtual int ReadElementContentAsBase64(byte[] buffer, int startIndex, int count);

public virtual int ReadElementContentAsBinHex(byte[] buffer, int startIndex, int count);

• Added ReadToFollowing(string localname, string namespaceURI) which moves to the next occurrence of the named element in document order.

XmlReaderSettings

• Added XmlSchemaValidationFlags enumeration to replace the following flags; IgnoreInlineSchema, IgnoreSchemaLocation, IgnoreValidationWarnings and IgnoreIdentityConstraints
• Added existing ValidationType enumeration to replace to replace the following flags; DtdValidate and XsdValidate

XmlWriter

• Reduced number of WriteValue() methods
• Removed overloads of WriteStartElement and WriteStartAttribute that took an IXmlSchemaInfo parameter

XPathDocument

• Reverted the XPathDocument to what it was in version 1.1 of the .NET Framework. It is no longer editable and no longer implements the IChangeTracking, IXmlSerializable, IXPathChangeNavigable, IXPathEditable and IRevertibleChangeTracking interfaces. The rationale for this decision is at More Information on the XPathDocument/XmlDocument Change in Whidbey beta 2

XPathNavigator and XPathEditableNavigator

• The XPathEditableNavigator has been merged into the XPathNavigator, making it an editable XML cursor model API.
• The XPathNavigator is the preferred API for exposing data as XML. This has been incorporated into the design guidelines for using XML in the .NET Framework

XmlDocument

• The XPathNavigator returned by the CreateNavigator() method now allows one to edit the XmlDocument through the cursor model API.
• The XmlDocument now supports XML schema validation of the entire subtree or partial validation of nodes in the document using the Validate() method
• The following property added to XmlDocument

public XmlSchemaSet Schemas { get; set; }

• The following property added to XmlNode

public virtual IXmlSchemaInfo SchemaInfo { get; }

XsltCommand

• The XslTransform class was obsoleted in Whidbey Beta 1 and replaced by the System.Xml.Query.XsltCommand class.  In Beta 2, we decided to revamp the XsltCommand API in order to make migration from XslTransform simpler.  This effort also resulted in the renaming of the XsltCommand class to  System.Xml.Xsl.XslCompiledTransform.
• XslCompiledTransform compiles XSLT to MSIL for significantly improved performance at the cost of increased (yet still small) compile times.
• Supports the MSXML XSLT extension functions such as format-date, format-time etc.

Inference

• This class has been renamed to XmlSchemaInference

XPathExpression -

• Added static Compile() method enables one to compile an input string containing an XPath query into an XPathExpression object

REMOVALS

XmlArgumentList

To reduce the cost of churn caused by the obsoletion of XslTransform this class has been removed. In its place the XsltArgumentList from v1.1 can be used

XQueryCommand

Microsoft has decided not to ship a client side XQuery implementation in .NET Framework 2.0 as our customers expect us to ship an implementation that meets the following criteria:

• Compliant with the W3C standards
• Functionally addresses key scenarios

As a core platform component in Windows, they also expect us to ship a product that meets the high bar of not breaking their applications when future updates are released.  After talking to key customers and partners, we have determined it is important that we cross this high bar before shipping a full implementation of XQuery in the platform. 

The best estimates tell us that ETA for XQuery to become a W3C recommendation is end of 2005 which does not fit with the .NET Framework 2.0 product release cycle.

In the meantime, we are shipping a well-defined small subset of XQuery in SQL Server 2005 to query information stored natively as XML data type.  This will enable new customer scenarios in SQL Server for storing and retrieving semi-structured data.

In the NET Framework 2.0 RTM timeframe, we recommend that our customers continue to use XSLT and XPath on the client side to solve their key client side filtering and transformation scenarios.  With this in mind, we have made significant improvements to our client side story including:

• Performance improvements - making the .NET Framework XSLT processor the best performing processor.
• Functional improvements - improving the usability and feature set of the existing .NET Framework processor

Note: As a result of not shipping XQuery, XML Views using mapping and XQuery to query SQL Server 2005 and the XmlAdapter to perform updates that were originally previewed in the PDC Alpha release of .NET V2.0 have also been removed. These were removed in the Beta 1 release.

 

I just finished writing last month's Extreme XML column* entitled The XML Litmus Test: Understanding When and Why to Use XML. The article is a more formal write up from my weblog post The XML Litmus Test expanded to contain examples of appropriate and inappropriate uses of XML as well as with some of the criteria for choosing XML fleshed out. Below is an excerpt from the article which contains the core bits that I hope everyone who reads it remembers  

XML is the appropriate tool for the job if the following criteria are satisfied by choosing XML as the data representation format for a given application.

1.      there is a need to interoperate across multiple software platforms

2.      one or more of the off-the-shelf tools for dealing with XML can be leveraged when producing or consuming the data

3.      parsing performance is not critical

4.      the content is not primarily binary content such as a music or image file

5.      the content does not contain control characters or any other characters that are illegal in XML

If the expected usage scenario does not satisfy most or all of the above criteria then it doesn't make much sense to use XML as the data representation format for the situation in question.

As the program manager responsible for XML programming models and schema validation in the .NET Framework I've seen lots and lots of inappropriate usage of XML both from internal teams and our customers. Hopefully once this article is published I can stop repeating myself and just send people links to it next time I see someone asking how to escape control characters in XML or see another online discussion of "binary" XML.

* Yes, it's late

 

In his post Debating WS-* Geoff Arnold writes

Tim Bray continues to discuss the relevance of the so-called WS-* stack: the collection of specifications related to XML-based web services. I'm not going to dive into the technology or business issues here; however Tim referred to a piece by Dare Obasanjo which argues that WS-* Specs are like JSRs. I tried to add a comment to this, but Dare's blog engine collapsed in a mess of XML, so I'll just post it here. Hopefully you'll be able to get back to read the original piece if you're interested. [Update: It looks as if my comment made it into Dare's blog after all.]

Just out of curiosity... if WS-* are like JSRs, what's the equivalent of the JCP? Where's the process documented, and what's the governance model? The statement "A JSR is basically a way for various Java vendors to standardize on a mechanism for solving a particular customer problem" ignores the fact that it's not just any old "way"; it's a particular "way" that has been publically codified, ratified by the community, and evolved to meet the needs of participants.

Microsoft isn't trying to compete with standards organizations. The JCP process falls out of the fact that Sun decided not to submit Java to a standards body but got pushback from customers and other Java vendors for something similar. So Sun manufactured an organization and process quite similar to a standards body with itself at the head. Microsoft isn't trying to get into this game.

The WS-* strategy that Microsoft is pursuing is informed from a lot of experience in the world of XML and standards. In the early days of XML, the approach to designing XML standards [especially at the W3C] was to throw together a bunch of preliminary ideas and competing draft specs without implementation experience then try to merge that into a coherent whole. This has been problematic as I wrote a few months ago

In recent times the way the W3C produces a spec is to either hold a workshop where different entities can submit proposals and then form a working group based on coming up with a unification of the various proposals or forming a working group to find come up with a unification of various W3C Notes  submitted by member companies. Either way the primary mechanism the W3C uses to produce technology specs is to take a bunch of contradictory and conflictiong proposals then have a bunch of career bureaucrats try to find some compromise that is a union of all the submitted specs. There are two things that fall out of this process. The first is that the process takes a long time, for example the XML Query workshop was in 1998 and six years later the XQuery spec is still a working draft. Also XInclude proposal was originally submitted to the W3C in 1999 but five years later it is just a candidate recommendation. Secondly, the specs that are produced tend to be too complex yet minimally functionaly since they compromise between too many wildly differing proposals. For example, W3C XML Schema was created by unifying the ideas behind DCD, DDML, SOX, and XDR. This has lead to a dysfunctional specification that is too complex for the simple scenarios and nigh impossible to use in defining complex XML vocabularies.

The WS-* process Microsoft has engaged the industry in aims at preventing this problems from crippling the world of XML Web Services as it has the XML world. Initial specs are written by the vendors planning who'll primarily be implementing the functionality then they are revised based on the results of various feedback and interoperability workshops. As a result of these workshops some specs are updated while others turn out to be infeasible and are deprecated. Some people such as Simon Fell, in his post WS-Gone, have complained that these leads to a situation where things are too much in flux but I think this is a lot better than publishing standards which turn out to contain features that are either infeasible to implement or are just plain wrong. Working in the world of XML technologies over the past three years I've seen both.

The intention is that eventually the specs that show that they are the fittest will end up in the standards process. This exactly what has happened with WS-Security (OASIS) and WS-Addressing (W3C). I expect more to follow in the future.

 

From Len Bullard

>What's the silver bullet?

It's a bar in Phoenix.

From Tim Bray

I disagree with virtually every technical argument Ted Nelson has ever
made and (in most cases) the implementations are on my side, but it
doesn't matter; Ted's place in history is secure because he asked more
important questions than just about anybody.   I think he usually
offered the wrong answers, but questions are more important.

The thread that produced these gems is Ted Nelson's "XML is Evil"  which revisits Ted Nelson's classic rant Embedded Markup Considered Harmful.

 

 

My article Improving XML Document Validation with Schematron is finally up on MSDN. It provides a brief introduction to Schematron, shows how to embed Schematron assertions in a W3C XML Schema document for improved validation capabilities and how to get the power of Schematron in the .NET Framework today. The introduction of the article is excerpted below

Currently the most popular XML schema language is the W3C XML Schema Definition language (XSD). Although XSD is capable of satisfying scenarios involving type annotated infosets it is fairly limited when it comes to describing constraints on the structure of an XML document. There are many examples of situations where common idioms in XML vocabulary design are impossible to express using the constraints available in W3C XML Schema. The three most commonly requested constraints that are incapable of being described by W3C XML Schema are:

1. The ability to specify a choice of attributes. For example, a server-status element should either have a server-uptime attribute or a server-downtime attribute.

2. The ability to group elements and attributes into model groups. Although one can group elements using compositors such as xs:sequence, xs:choice, and xs:all, the same cannot be done with both elements and attributes. For example, one cannot create a choice between one set of elements and attributes and another.

3. The ability to vary the content model based on the value of an element or attribute. For example, if the value of the status attribute is "available" then the element should have an uptime child element; otherwise it should have a downtime child element. The technical name for such constraints is co-occurrence constraints.

Although these idioms are widely used in XML vocabularies it isn't possible to describe them using W3C XML Schema, which makes it difficult to rely on schema validation for enforcing the message contract. This article describes how to layer such functionality on top of the W3C XML Schema language using Schematron.

Embedding Schematron assertions in a W3C XML Schema document allows you to have your cake and eat it to.

 

Tim Bray has another rant on the prolifieration of WS-* specs in the XML Web Services world. In his post The Loyal WS Opposition he writes

I Still Dont Buy It No matter how hard I try, I still think the WS-* stack is bloated, opaque, and insanely complex. I think its going to be hard to understand, hard to implement, hard to interoperate, and hard to secure.

I look at Google and Amazon and EBay and Salesforce and see them doing tens of millions of transactions a day involving pumping XML back and forth over HTTP, and I cant help noticing that they dont seem to need much WS-apparatus.

One way to view the various WS-* specifications is that they are akin to Java Specification Requests (JSRs) in the Java world. A JSR is basically a way for various Java vendors to standardize on a mechanism for solving a particular customer problem. Usually this mechanism takes the form of an Application Programming Interface (API). Some JSRs are widely adopted and have become an integral aspect of programming on the Java platform (e.g. the JAXP JSR). Some JSRs are pushed by certain vendors while being ignored by others leading to overlap (e.g. the JDO JSR which was voted against by BEA, IBM and Oracle but supported by Macromedia and Sun). Then there's Enterprise Java Beans which is generally decried as a bloated and unnecessarily complex solution to business problems. Again that was the product of the JSR process.

The various WS-* specs are following the same pattern as JSRs which isn't much of a surprise since a number of the players are the same (e.g. Sun & IBM). Just like Tim Bray points out that one can be productive without adopting any of the WS-* family of specifications it is similarly true that one can be productive in Java without relying on the products of JSRs but instead  rolling one's own solutions. However this doesn't mean there aren't benefits of standardizing on the high level mechanisms for solving various business problems besides saying "We use XML and HTTP so we should interop".   

Omri Gazitt, the Produce Unit Manager of the Advanced XML Web Services team has a post on WS-Transfer and WS-Enumeration which should hit close to home for Tim Bray since he is the co-chair of the Atom working group

WS-Transfer is a spec that Don has wanted to publish for a year now.  It codifies the simple CRUD pattern for Web services (the operations are named after their HTTP equivalents - GET, PUT, DELETE, and there is also a CREATE pattern.  The pattern of manipulating resources using these simple verbs is quite prevalent (Roy Fielding's REST is the most common moniker for it), and of course it underlies the HTTP protocol.  Of course, you could implement this pattern before WS-Transfer, but it does help to write this down so people can do this over SOAP in a consistent way.  One interesting existing application of this pattern is Atom (a publishing/blogging protocol built on top of SOAP).  Looking at the Atom WSDL, it looks very much like WS-Transfer - a GET, PUT, DELETE, and POST (which is the CREATE verb specific to this application).  So Atom could easily be built on top of WS-Transfer.  What would be the advantage of that?  The same advantage that comes with any kind of consistent application of a technology - the more the consistent pattern is applied, the more value it accrues.  Just the value of baking that pattern into various toolsets (e.g. VS.NET) makes it attractive to use the pattern. 

I personally think WS-Transfer is very interesting because it allows SOAP based applications to model themselves as REST Web Services and get explicit support for this methodolgy from toolkits. I talked about WS-Transfer with Don a few months ago and I've had to bite my tongue for a while whenever I hear people complain that SOAP and SOAP based toolkits don't encourage building RESTful XML Web Services.

I'm not as impressed with WS-Enumeration but I find it interesting that it also covers another use case of the ATOM API which is a mechanism for pulling down the content archive  from a weblog or similar system in a sequential manner. 

 

Oleg has just announced a new release of EXSLT, his post is excerpted below

Here we go again - I'm pleased to announce EXSLT.NET 1.1 release. It's ready for download. The blurb goes here:

EXSLT.NET library is community-developed free open-source implementation of the EXSLT extensions to XSLT for the .NET platform. EXSLT.NET fully implements the following EXSLT modules: Dates and Times, Common, Math, Random, Regular Expressions, Sets and Strings. In addition EXSLT.NET library provides proprietary set of useful extension functions.

Download EXSLT.NET 1.1 at the EXSLT.NET Workspace home - http://workspaces.gotdotnet.com/exslt
EXSLT.NET online documentation - http://www.xmland.net/exslt

EXSLT.NET Features:

• 65 supported EXSLT extension functions
• 13 proprietary extension functions
• Support for XSLT multiple output via exsl:document extension element
• Can be used not only in XSLT, but also in XPath-only environment
• Thoroughly optimized for speed implementation of set functions

Here is what's new in this release:

• New EXSLT extension functions has been implemented: str:encode-uri(), str:decode-uri(), random:random-sequence().
• New EXSLT.NET extension functions has been implemented: dyn2:evaluate(), which allows to evaluate a string as an XPath expression, date2:day-name(), date2:day-abbreviation(), date2:month-name() and date2:month-abbreviation() - these functions are culture-aware versions of the appropriate EXSLT functions.
• Support for time zone in date-time functions has  been implemented.
• Multithreading issue with ExsltTransform class has been fixed. Now ExsltTransform class is thread-safe for Transform() method calls just like the  System.Xml.Xsl.XslTransform class.
• Lots of minor bugs has been fixed. See EXSLT.NET bug tracker for more info.
• We switched to Visual Studio .NET 2003, so building of the project has been greatly simplified.
• Complete suite of NUnit tests for each extension function has been implemented (ExsltTest project).

The EXSLT.NET project has come quite some way since I started it last year. Oleg has done excellent work with this release. It's always great to see the .NET Open Source community come together this way.

 

Roger Costello recently started a discussion thread on the XML-DEV mailing list about the common misconceptions people have about XML document validation and schemas. He recently summarized the discussion thread in his post Fallacies of Validation, version #3. His post begins

The purpose of documenting the below "fallacies" is to identify erroneous common thought that many people have with regards to validation and its role in a system architecture.  Perhaps "assumptions" would be a better term to use than "fallacies".  In any case, the desire of this writeup (which is a compilation of discussions on the xml-dev list) is to provoke new ways of thinking about validation, and reject limiting and static views on validation. 

Fallacies of Validation

1. Fallacy of "THE Schema"

2. Fallacy of Schema Locality

3. Fallacy of Requisite Validation

4. Fallacy of Validation as a Pass/Fail Operation

5. Fallacy of a Universal Validation Language

6. Fallacy of Closed System Validation

7. Fallacy that Validation is Exclusively for Constraint Checking

I mostly agree with the fallacies as described in his post.

Fallacy #1 has been a favorite topic of Tim Ewald over the past year. It isn't necessarily true that there is one canonical schema for an XML vocabulary. Instead the schema for the vocabulary may depend on the context the XML document is being used in. A classic example of this is XHTML which has 3 schemas (DTDs) for a single format.

I consider Fallacy #2 to be more of a common mistake than a fallacy. Many people create validation systems that work in a local environment such as creating specific patterns or structures for addresses or telephone numbers which may work in a local system but break down when used in a global environment like the World Wide Web. This common mistake isn't limited to XML validation but applies to all arenas where user input is validated before being stored or processed

Fallacy #3 is interesting to me because I wonder how often it occurs in the wild. Are there really that many people who believe they have to validate XML documents against a schema?

Fallacy #4 is definitely a good one. However I disagree with the quotes he uses to butress the main point for this fallacy. I especially don't like the fact that he uses a generalization from Rick Jellife about bugs in a few schema validators as a core part of his argument. The important point is that schema validation should not always be viewed as a PASS/FAIL operation and in fact schema languages like W3C XML Schema go out of their way to define how one can view an XML document as being part valid, part invalid.

One size doesn't fit all is the message of Fallacy #5 to which I heartily cheer "Hear! Hear!". I agree 100%. There is no one XML schema language that satisfies every validation scenario.

I don't really understand Fallacy #6 without seeing some examples so I won't comment on it. I'll see if I can dig up the discussion threads about this on XML-DEV later.

Fallacy #7 is another one where I agree with the message but mostly disagree with how he argues the point. All of his examples are all variations of using schemas for constraint checking, they just differ on how the document is processed does after constraint checking is done. To me, the prime example of the fact that schema validation is not just for constraint checking is that many technologies actually using schemas for creating typed XML documents or for translating XML from one domain to another (e.g. Object<->XML, Relational<-> XML),

Everything said, this was a good list. Excellent work from Roger as usual.

 

In my recent post entitled The MSDN Camp vs. The Raymond Chen Camp I wrote

Our team [and myself directly] has gone through a process of rethinking a number of decisions we made in this light. Up until very recently we were planning to ship the System.Xml.XPath.XPathDocument class as a replacement for the System.Xml.XmlDocument class
...
The problem was that the XPathDocument had a radically different programming model than the XmlDocument meaning that anyone who'd written code using the XmlDocument against our v1.0/v1.1 bits would have to radically rewrite their code to get performance improvements and new features. Additionally any developers migrating to the .NET Framework from native code (MSXML) or from the Java world would already be familiar with the XML DOM API but not the cursor-based model used by the XPathDocument. This was really an untenable situation. For this reason we've reverted the XPathDocument to what it was in v1.1 while new functionality and perf improvements will be made to the XmlDocument. Similarly we will keep the new and improved XPathEditableNavigator XPathNavigator class which will be the API for programming against XML data sources where one wants to abstract away what the underlying store actually is. We've shown the power of this model with examples such as the ObjectXPathNavigator and the DataSetNavigator.

I've seen some concerned statements about this posts from XML developers who use System.Xml such as Oleg Tkachenko, Fumiaki Yoshimatsu and Tomas Restrepo so it seems I should clarify some of the decisions we made and why we made them.

In version 1.0 of the .NET Framework we provided two primary classes for interacting with XML; the XmlDocument and XmlReader. The XmlReader provided an abstract interface for interacting with a stream of XML. One can create an XmlReader over textual XML using the XmlTextReader or over virtual XML data sources such as is done with the XmlCsvReader. On the other hand with the XmlDocument we decided to eschew the approach favored by the Java world which used interfaces. Instead we created a single concrete implementation. This turned out to be a bad idea. It tied the the interface for programming against XML in a random access manner with a concrete implementation of an XML store. This made it difficult for developers who wanted to expose their data sources as XML stores and led to inefficient solutions such as the XmlDataDocument.

To rectify this we needed to separate the programming model for accessing XML data sources from our concrete implementation of the XmlDocument. We chose to do this by extending the cursor based programming model we introduced in v1 with the XPathNavigator instead of moving to an interface based approach with XmlDocument. The reason for choosing to go with a cursor based model over a tree based model is summed up in this quote from my article Can One Size Fit All?

In A Survey of APIs and Techniques for Processing XML, I pointed out that cursor-model APIs could be used to traverse in-memory XML documents just as well as tree-model APIs. Cursor-model APIs have an added advantage over tree-model APIs in that an XML cursor need not require the heavyweight interface of a traditional tree-model API where every significant token in the underlying XML must map to an object.

So in Whidbey, the XPathNavigator will be the programming model for working with XML data sources when one wants to abstract away from the underlying source. The XPathNavigator will be changed from the v1.0 model in the following ways (i) it will be editable and (ii) it will expose the post schema validation infoset. I've already worked with Krzysztof Cwalina on updating the Design Guidelines for Exposing XML data in WinFX to account for this change in affairs.

As for the XPathDocument, it is what it always has been. A class optimized for use in XPath and XSLT. If you need 10% - 25% better perf [depending on your scenario] when running XPath over an XML document or running XSLT over in-memory XML then this class should be preferred to the XmlDocument.   

 

A few months ago in Joel Spolsky's  How Microsoft Lost the API War  he wrote

There are two opposing forces inside Microsoft, which I will refer to, somewhat tongue-in-cheek, as The Raymond Chen Camp and The MSDN Magazine Camp.
...
The Raymond Chen Camp believes in making things easy for developers by making it easy to write once and run anywhere (well, on any Windows box). The MSDN Magazine Camp believes in making things easy for developers by giving them really powerful chunks of code which they can leverage, if they are willing to pay the price of incredibly complicated deployment and installation headaches, not to mention the huge learning curve. The Raymond Chen camp is all about consolidation. Please, don't make things any worse, let's just keep making what we already have still work. The MSDN Magazine Camp needs to keep churning out new gigantic pieces of technology that nobody can keep up with.

When I first read the above paragraphs I disagreed with them because I was in denial. But as the months have passed and I've looked at various decisions my team has made in recent years I see the pattern. The patterns repeats itself in the actions of other product teams and divisions at Microsoft. I know realize this is an unfortunate and poisonous aspect of Microsoft's culture which doesn't work in the best interest of our customers. A few months ago I found some advice given to Ward Cunningham on joining Microsoft which read

 Take a running start and don't look back

1. Recognize that your wonderful inventiveness is the most valuable thing you will own in a culture that values its employees solely by their latest contributions. In a spartan culture like this, you will rise quickly.

2. Keep spewing ideas, even when those ideas are repeatedly misunderstood, implemented poorly, and excised from products for reasons that have nothing to do with the quality of the idea. When you give up on communicating your new ideas, you will just go insane waiting to vest.

The Microsoft culture is about creating the newest, latest greatest thing that 'changes the world' not improving what is already out there and working for customers. When I read various Microsoft blogs and MSDN headlines about how even though we've made paradigm shifts in developer technologies in the recent years we aren't satisfied and want to introduce radically new and different technologies all over again. This bothers me. I hate the fact that 'you have to rewrite a lot of your code' is a common answer to questions a customer might ask about how to leverage new or upcoming functionality in a developer technology.

Our team [and myself directly] has gone through a process of rethinking a number of decisions we made in this light. Up until very recently we were planning to ship the System.Xml.XPath.XPathDocument class as a replacement for the System.Xml.XmlDocument class. One of the driving reasons for doing this was XPath and XSLT performance. The mismatch between the DOM data model and that of XPath meant that XPath queries or XSLT transformations over the XmlDocument would never be as fast as XPathDocument. Another reason we were doing this was that since the XmlDocument is not an interface based design there isn't a way for people who implement their own XML document-like classes to plug-in to our world. So we decided to de-emphasize (but not deprecate) the XmlDocument by not adding any new functionality or performance improvements to it and focused all our energy on XPathDocument.

The problem was that the XPathDocument had a radically different programming model than the XmlDocument meaning that anyone who'd written code using the XmlDocument against our v1.0/v1.1 bits would have to radically rewrite their code to get performance improvements and new features. Additionally any developers migrating to the .NET Framework from native code (MSXML) or from the Java world would already be familiar with the XML DOM API but not the cursor-based model used by the XPathDocument. This was really an untenable situation. For this reason we've reverted the XPathDocument to what it was in v1.1 while new functionality and perf improvements will be made to the XmlDocument. Similarly we will keep the new and improved XPathEditableNavigator XPathNavigator class which will be the API for programming against XML data sources where one wants to abstract away what the underlying store actually is. We've shown the power of this model with examples such as the ObjectXPathNavigator and the DataSetNavigator.

It's good to be back in the Raymond Chen camp.

 

Every once in a while I like to post a list of articles I'm either in the process of writing or considering writing to get feedback from people on what they'd like to see or whether the topics are even worthwhile. Below is a list of the next couple of articles I'm either in the process of writing or plan to write over the next few months.

1. An Introduction to Validating XML Documents with Schematron (MSDN) : An introduction to Schematron including examples showing how one can augment a W3C XML Schema document using Schematron thus creating an extremely powerful XML schema language.  Code samples will use Schematron.NET

2. Designing XML Formats: Versioning vs. Extensibility (XML 2004 Conference) : This is the presentation and paper for my XML 2004 talk. It will basically be the ideas in my article On Designing Extensible, Versionable XML Formats with more examples and less fluff.

3. The XML Litmus Test - Deciding When and Why To Use XML (MSDN) : After seeing more and more people at work who seem to not understand what XML is good for or what the decision making process should be for adopting XML I decided to put this article together.  This will basically be an amalgamation of my XML Litmus Test blog post and my Understanding XML article on MSDN.  

4. XML in Cw (XML.com)  : An overview of the XML based features of Cw. The Cw type system contains several constructs that reduce the impedance mismatch between OO and XSD by introducing concepts such as anonymous types, choices [aka union types], nullable types and constructing classes from XML literals into the .NET world. The ability to process such strongly typed XML objects using rich query constructs based on SQL's select operator will also be covered.

5. A Comparison of Microsoft's C# Programming Language to Sun Microsystems' Java Programming Language 2nd edition : About 3 years ago I wrote a C# vs. Java comparison while I was still in school which has become the most popular comparison of both languages on the Web. I still get mail on a semi-regular basis from people who've been able to transition between both languages due to the information in my comparison document. I plan to update this article to reflect the proposed changes announced in Java 1.5 and C# 2.0

On top of this I've been approached twice in the past few months about writing a technology book. Based on watching the experiences of others my gut feel is that it isn't worth the effort. I'd be interested in any feedback on the above article ideas or even suggestions for new articles that you'd be interested in seeing on MSDN or XML.com from me.

 

In his post I Want RELAX NG! Tim Ewald writes

This recent post on Mark Nottingham's site pushed me over the edge. I agree with Sean's comment: I want Relax NG. Can I make systems work with XSD? Yes, sort of. But it adds a ludicrous amount of complexity. First you have to know how it works, then what not to do because it's too complicated (like complicated type or element substitution models), then figure out how to contort your schema to do what you want (like extensibility and versioning). Relax NG is much simpler and much closer to how XML actually works. And yes, you can still map it to /from objects if you want to.

I can't help but wonder why, if WS-* and SOAP 1.2 keep XSD at arms length (referencing simple types only and providing non-normative schema definitions) and WSDL 2.0 defines its own simple types, everyone assumes I want to use XSD to define my Web service interface. Pretty much everyone I know who works in this space agrees that Relax NG is a better choice. What is stopping us from making this change?

This is one of those times where I both agree and disagree with Tim. To explain why, I first need to list the two three reasons people tend to write schemas.

1. To provide a way to annotate an XML document with type information and thus created a type annotated infoset.
2. To provide a means to ensure that an XML documents satisfies the constraints of a given message contract
3. To provide terse, human readable documentation of an XML format.

In most developer scenarios [including XML Web Services] the most popular use case is the first from the list above. An XML Schema is used primarily for mapping the contents of an XML document either into relational tables (e.g. SQLXML, ADO.NET DataSet) or into a set of programming language objects (e.g. System.Xml.Serialization.XmlSerializer). Every XML Web Service toolkit I have encountered emphasizes this scenario and in fact most customers do not use XML schemas for validation of business documents for either performance reasons or the fact that their business rules cannot be adequately described using an XML schema. The main problem with XSD for this use case is that it is actually too expressive and has a richer type system than either the relational model or traditional object oriented programming languages. This leads to impedance mismatches which makes it hard for XML Web Service stacks to map schema declarations to objects thus leading to calls from folks like the WS-I to propose creating a subset or profile of XSD.

On the other hand, XSD is notoriously bad at dealing with the second use case described above. The language makes either makes it hard to describe common XML idioms (see the hoops I have to jump through in my Designing Extensible, Versionable XML Formats article) or impossible (e.g. if an attribute has a certain value then the element should have a certain content model or the providing a choice of attributes). This is where RELAX NG shines. Of course, being more expressive than XSD means that the impedance mismatch between it and the relational and OO models is even more significant. 

In practice today, most XML Web Services need an XML schema language for creating type annotated infosets not for validating message structure. This means that for their use cases XSD is preferable to RELAX NG. Ideally, a simple language that just allowed creating named structures and primitive types such as Microsoft's now-obsolete XML Data Reduced (XDR) would be even more optimal.  

Of course, the XML Web Services world could one day evolve to the point where being able to validate incoming messages against a schema is deemed more important than being able to deserialize the XML into objects and vice versa. In which case, Aaron Skonnard's statement in his post Could RelaxNG Replace XSD? which describe the existing industry inertia around XSD is also a point to consider.

 

I saw the following excerpt in Shelley Powers's post entitled Differences of Humor where she wrote

Sam Ruby has posted a note about the upcoming Applied XML Conference put on by Chris Sells. When I looked at the agenda and realized that the conference managed to put together two days worth of presentations without one woman speaker,

Knowing the nature of Chris Sells's conferences this is unsurprising. They seem to mostly be an opportunity for Chris's DevelopMentor clique and their buddies to hang out. However Shelley's post did make me start thinking about how many women I knew who worked with XML and just like the time I started to keep a list of Seinfeld episodes in which at least one African or African American appeared in (don't ask) I started tracking down the number of women I knew off who worked on XML technologies who's works I'd rather see present than at least one of the presentations currently on the roster. Here is my list

Non-Microsoft

• Eve Maler - Sun's most notable XML geek after Jon Bosak and Tim Bray. She's worked on SAML and UBL. I meet her at XML 2003 where we chatted about versioning in UBL and what truly meant by polymorphic XML processing.

• Jeni Tennison - the most knowledgeable person on the planet about W3C XML Schema. I've lost count of the amount of times I've seen her school members of the W3C XML Schema working group about the technology on various mailing lists. Also an XSLT and XPath guru. She's always pushing boundaries in the XML world such as with her work on layered hierarchies in markup vocabularies with LMNL

• Priscilla Walmsley - the author of Definitive XML Schema which is probably the best book on W3C XML Schema on the market. She's also co-written a book on XML in Office 2003 which I haven't read but would love to get a presentation on especially with regard to some finer details on how Office uses XML schemas. 

• Amelia Lewis - a co-author of the WS-ReliableMessaging specification and the author of an excellent critique of the W3C XML Schema primitive types in her article Not My Type: Sizing Up W3C XML Schema Primitives
  

Microsoft

• Elena - the Microsoft XML Web Service stack rests on her shoulders. What makes Visual Studio.NET an awesome XML Web Service environment is that there is functionality that lets you point at a WSDL and automatically you get handy dandy .NET classes generated for you. Elena owns the meat of this code, a lot of which resides in the XmlSerializer class

• Denise Draper - an architect on our team who in a past life has been a member of the XQuery working group, worked an XML data integration suite for Nimble Technology and worked in the AI field.

• Priya Lakshminarayanan - the developer for the W3C XML Schema validation technology in the .NET Framework. She's the most knowledgeable about the technology at Microsoft, I'm a distant second to her breadth of knowledge about this somewhat arcane and cryptic technology. She the first person I've seen implement a tool for generating sample XML documents from XML schemas that didn't suck.

• Helena Kupkova - the developer for the XML parser in the .NET Framework. She completely gutted our old implementation and doubled the perf in some scenarios. A totally impressive developer. More impressive is that she ships stuff like the XML Diff and Patch demo on GotDotNet in her spare time.

• Nithya Sampathkumar - the developer on the XML schema inference technology in the .NET Framework. Once I took over as the program manager for this technology I grew to understand the subtleties involved in trying to infer a schema for arbitrary XML documents. A presentation on the techniques used in her implementation and the limitations of XML schema inference would be quite interesting.

• Neetu Rajpal - the program manager for XML tools in Visual Studio. I've overheard some interesting conversations involving her discussing some of the trickiness involved in implementing an XSLT debugger. An in-depth presentation about what the XML tools team is planning to ship and the issues they encountered would be killer.

• Vinita - the program manager for MSXML which is the most widely deployed XML library on the planet. Even without shipping in Internet Explorer, Windows and Office they still get millions of downloads a year.

• Tejal Joshi - works on the XML tools in Visual Studio. At last year's XML 2003 conference I enjoyed hearing James Clark discuss implementation strategies for his nxml-mode in Emacs. I'm sure Tejal would have similarly interesting stories to tell.

• Lanqing Dai - used to be developer for the XmlDocument class but has moved on to WinFS. I'd love to hear a her thoughts on how working in an XML-centric world compares to living in the item-centric world of WinFS.

There are more women I know off in the XML field both within and outside Microsoft but these are the ones whose presentations I'd rather see than something like XML as a Better COM (for example). Maybe next time Chris Sells should look around the usual XML hang outs both online (like the xml-dev mailing list) and within Microsoft internally for conference speakers instead of announcing them in his blog. It may lead to a more diverse list of topics and speakers.

I need to go watch Berserk. Talk to you guys later.

 

According to the current version of the Chris Sells XML DevCon page (don't bother bookmarking it, Chris Sells doesn't believe in permalinks so all the content on that page is transient) I noticed that Chris Anderson is presenting the following

Developers Hate XML

Chris Anderson

While everyone is currently infatuated with XML, developers are constantly doing battle with trying to rationalize and leverage XML in their applications. Ill talk about having to balance correct XML-isms vs. usability in XAML, about the preponderance of XML reader/writer/DOM/serialization APIs, and about how all of this throws you into a horrible programming experience of loosely typed runtime errors. This reveals XML for what it is a data encoding. XML is the ASCII text file of the 2000s. While web services are often called "XML Web Services," the reality is that every web service API abstracts the developer from the XML view.

Nothing says vote of confidence like when the chief architects of one of teams you work closely with says your technology sucks. :)

Seriously though, I am curious as to what his presentation actually will be about. Reading the abstract, it seems like it is another iteration of a data-centric user of XML coming to the realization that for their scenarios XML is just CSV on steroids. People's behavior when they realize this usually follows a pattern similar to the five stages of grief. First there is denial, this usually takes the form of an initial disbelief that after all the hype they've heard about XML it isn't working out fantastically for them. Then there is bargaining, this usually manifests itself as attempts to not use XML but still use it. Often you here phrases like "binary XML", "XML subset" or "XML profile" at this point. Then there is anger at XML for being more complex and verbose than they need. At this point you get to either read a rant-filled email, blog post, conference paper or in this case conference presentation about how badly the technology is suited for its purpose. Then there is either despair or acceptance. One doesn't follow the other. If the next stage is despair in this case the person ends up not using XML to solve that particular problem. On the other hand if it is acceptance, XML is still used but in some cases it is in one of the forms that were mentioned in the bargaining stage such as a binary representation of an XML stream or uing some subset of XML.

Hopefully Chris Anderson will post his slides online.

 

I have been remiss about talking about the ongoing and upcoming content on the MSDN XML Developer Center. The most interesting recent article has been Priya's Generating XML Documents from XML Schemas. In this article she provides a tool that allows you to generate sample documents that validate against a particular schema. This is very useful if you have a schema document but would like to see what a valid instance of that schema looks like. This is one of those tools I'd love to see added to the XML editor in Visual Studio. The one caveat is that her tool requires .NET Framework v2.0 beta 1 or higher to run.

The next couple of articles we have scheduled include an overview of how to use EXSLT to make one more productive as an XSLT developer by Oleg Tkachenko, an implementation of XPathNavigator over the ADO.NET DataSet by Arpan Desai, and an introduction Daniel Cazzulino's implementation of Schematron for the .NET Framework by myself.

I'm currently working on the content plan for the next quarter of the year and would like to know what articles people have liked and would like to see in the future. Also if you are interested in writing for MSDN about XML technologies on Microsoft platfortms go ahead and send me an email at my work address.

 

On July 8th, a couple of us from the XML team held a hosted chat on MSDN Chats. The transcripts of the C# and XML chat are now available. We answered questions on our existing behavior in Everett as well as upcoming technologies in Whidbey. If there are any followup questions to those asked during the chat just post them here and I'd love to answer them.
 

I recently got a number of bug reports that in certain situations RSS Bandit would report a proxy authentication error when fetching certain RSS feeds over the Web when connecting through a proxy server. It seemed most feeds would work fine but a particular set of feeds would result in the following message

The remote server returned an error: (407) Proxy Authentication Required.

Examples of sites that had problems include the feeds for Today on Java.net, Martin Fowler's bliki and Wired News. It dawned on me that the one the one thing all these feeds had in common was that they referenced a DTD. The problem was that although I was using an instance of the System.Net.IWebProxy interface in combination with an HttpWebRequest when fetching the RSS feed I did not provide the XmlValidatingReader used to process the feed that it should use the proxy information when resolving DTDs.  

This is where things got less intuitive. All XmlReaders have an XmlResolver property used to retrieve resources external to the file. However the XmlResolver class does not provide a way to specify proxy information, only authenticattion information. To solve this problem I had to create a subclass of the XmlResolver class which used the proxy connection when retrieving external resources. It seems I'm not the only person who's come up across this problem and the solution was presented on the microsoft.public.dotnet.xml newsgroup a while ago in the thread entitled XmlValidatingReader, XmlResolver, Proxy Authentication, Credentials, Remote schema. This post shows how to create a custom XmlResolver which utilizes proxy information and how to use this class to prevent the errors I was seeing.

I checked in the fix to RSS Bandit this morning, so very soon a number of users of the most sophisticated news aggregator on the Windows platform will be very happy campers seeing this annoying bug fixed.  

 

About a week ago my article Designing Extensible, Versionable XML Formats appeared on XML.com. However due to a “pilot error” on my end I didn't send the final draft to XML.com. By the time I realized my mistake the article was already live and changing it would have been cumbersome since there were a few major changes in the article.

You can read the final version of the article Designing Extensible, Versionable XML Formats on MSDN. The main differences between the MSDN article and the XML.com one are

1. Added sections on Message Transfer Negotiation vs. Versioning Message Payloads and Version Numbers vs. Namespace Names

2. Added more content to the section Using XML Schema to Design an Extensible XML Format especially around usage of substitution groups, xsi:type and xs:redefine.

3. Amended all sample schemas to use blockdefault="#all".

4. Added an Acknowledgements section

5. Schema in for section New constructs in a new namespace approach uses a fixed value instead of a default value for mustUnderstand attribute on isbn element.

 

Today Arpan (the PM for XML query technologies in the .NET Framework) and I were talking about features we'd like to see on our 'nice to have' list for the Orcas release of the .NET Framework. One of the things we thought would be really nice to see in the System.Xml namespace was XPath 2.0. Then Derek being the universal pessimist pointed out that we already have APIs that support XPath 1.0 that only take a string as an argument (e.g. XmlNode.SelectNodes) so we'd have difficulty adding support for another version of XPath without contorting the API.

Not to be dissuaded I pointed out that XPath 2.0 has a backwards compatibility mode which makes it compatible with XPath 1.0. Thus we wouldn't have to change our Select methods or introduce new methods for XPath 2.0 support since all queries that used to work in the past against our Select methods would still work if we upgraded our XPath implemention to version 2.0. This is where Arpan hit me with the one-two punch. He introduced me to a section of the XPath 2.0 spec called Incompatibilities when Compatibility Mode is true which reads

The list below contains all known areas, within the scope of this specification, where an XPath 2.0 processor running with compatibility mode set to true will produce different results from an XPath 1.0 processor evaluating the same expression, assuming that the expression was valid in XPath 1.0, and that the nodes in the source document have no type annotations other than xdt:untypedAny and xdt:untypedAtomic.

I was stunned by what I read and I am still stunned now. The W3C created XPath 2.0 which is currently backwards incompatible with XPath 1.0 and added a compatibility mode option to make it backwards compatible with XPath 1.0 but it actually still isn't backwards compatible even when in this mode?  This seems completely illogical to me. What is the point of having a backwards compatibility mode if it isn't backwards compatible? Well, I guess now I know if we do decide to ship XPath 2.0 in the future we can't just add support for it transparently to our existing classes without causing some API churn. Unfortunate.

Caveat: The fact that a technology is mentioned as being on our 'nice to have' list or is suggested in a comment to this post is not an indication that it will be implemented in future versions of the .NET Framework.

 

I was reading an XML-Deviant column on XML.com entitled Browser Boom when I came across the following excerpt

The inevitable association with Microsoft's CLI implementation is proving a source of difficulty for the Mono project. The principal author of Mono's XML support, Atsushi Eno, posted to the Mono mailing list on the problems of being conformant in Mono's XML parser implementation. More specifically, whose rules should Mono conform to. W3C or Microsoft?

MS XmlTextReader is buggy since it accepts XML declaration as element content (that violates W3C XML specification section 3 Logical Structures). ... However, there is another discussion that it is useful that new XmlTextReader (xmlText, XmlNodeType.Element, null) accepts XML declaration.

... that error-prone XmlTextReader might be useful (especially for people who already depends on that behavior)

... we did not always reject Microsoft badness; for example we are copying System.Xml.XmlCDataSection that violates W3C DOM interface hierarchy (!)

The root of the dilemma is similar to that which Mozilla and Opera are trying to manage in the browser world.

What I find interesting is that instead of pinging the MSFT XML folks (like myself) and filing a bug report this spawned a dozen message email discussion on whether Mono should be bug compatible with the .NET Framework. Of course, if the Mono folks decide to be bug compatible with this and other bugs in System.Xml and we fix them thus causing breaking changes in some cases will we see complaints about how Microsoft is out to get them by being backwards incompatible? Now that Microsoft has created the MSDN Product Feedback Center they don't even have to track down the right newsgroup or email address of a Microsoft employee to file the bug.

It's amazing to me how much work people cause for themselves and conspiracy theories they'd rather live in than communicate with others.

Update: I talked to developer responsible for the XmlTextReader class and she responded "This is by design. We allow XML declaration in XML fragments because of the encoding attribute. Otherwise the encoding information would have to be transferred outside of the XML and manually set into XmlParserContext."

 

A little while ago some members of our team experimented various ways to reduce the Relational<->Objects<->XML (ROX) impedance mismatch by adding concepts and operators from the relational and XML (specifically W3C XML Schema) world into an object oriented programming language. This effort was spear headed by a number of smart folks on our team including Erik Meijer, Matt Warren, Chris Lovett  and a bunch of others all led by William Adams. The object oriented programming language which was used as a base for extension was C#. The new language was once called X# but eventually became known as Xen.

Erik Meijer presented Xen at XML 2003 and I blogged about his presentation after the conference. There have also been two papers published about the ideas behind Xen; Programming with Rectangles, Triangles, and Circles and Unifying Tables, Objects and Documents. It's a new year and the folks working on Xen have moved on to other endeavors related to future versions of Visual Studio and the .NET Framework.

However Xen is not lost. It is now part of the Microsoft Research project, Cw (pronounced C-Omega). Even better you can download a preview of the Cw  compiler from the Microsoft Research downloads page

 

In a post entitled Dare Obasanjo is raining on the W3C's parade, Mike Dierken responds to my recent post which asks Is the W3C Becoming Irrelevant? by writing

Either way the primary mechanism the W3C uses to produce technology specs is to take a bunch of contradictory and conflictiong proposals then have a bunch of career bureaucrats try to find some compromise that is a union of all the submitted specs

Damn those career bureaucrats that built XML. Or is it the SOAP design process that caused the grief? And where did that technology come from anyway?

My original post already described the specs that have caused grief and show the W3C is losing its way. I assume that Mike is trying to use XML 1.0 and SOAP 1.1 as counter examples to the trend I pointed out. Well first of all, XML 1.0 was a proposal to design a subset of SGML so by definition it could not suffer the same problems that face attempts to innovate by committee which have hampered the W3C in current times. Also when XML 1.0 was created it was much smaller and a majority of the participants in the subsetting of SGML had similar goals. As for SOAP 1.1, it isn't a W3C spec. SOAP 1.1 was created by Don Box, Dave Winer and a bunch of Microsoft and IBM folks and then submitted to the W3C as a W3C Note.

Of course, the W3C has created iterations of both specs (XML 1.1 & SOAP 1.2) which in both cases are backwards incompatible with the previous versions. I leave it as an excercise to the reader to decide if having backwards incompatible point releases of Web specifications is how one 'leads the Web to its full potential'.

 

For a long time I used to think the W3C held the future of the World Wide Web in its hands. However I have come to realize that although this may have been true in the past the W3C has become too much of a slow moving bureaucratic machine to attract the kind of innovation that will create the next generation of the World Wide Web. From where I sit there are three major areas of growth for the next generation of the World Wide Web; the next generation of the dynamic Web, syndication and distibuted computing across the Web. With the recent decisions of Mozilla and Opera to form the WHAT working group and Atom's decision to go with the IETF it seems the W3C will not be playing a dominant role in any of these 3 areas.

In recent times the way the W3C produces a spec is to either hold a workshop where different entities can submit proposals and then form a working group based on coming up with a unification of the various proposals or forming a working group to find come up with a unification of various W3C Notes  submitted by member companies. Either way the primary mechanism the W3C uses to produce technology specs is to take a bunch of contradictory and conflictiong proposals then have a bunch of career bureaucrats try to find some compromise that is a union of all the submitted specs. There are two things that fall out of this process. The first is that the process takes a long time, for example the XML Query workshop was in 1998 and six years later the XQuery spec is still a working draft. Also XInclude proposal was originally submitted to the W3C in 1999 but five years later it is just a candidate recommendation. Secondly, the specs that are produced tend to be too complex yet minimally functionaly since they compromise between too many wildly differing proposals. For example, W3C XML Schema was screated by unifying the ideas behind DCD, DDML, SOX, and XDR. This has lead to a dysfunctional specification that is too complex for the simple scenarios and nigh impossible to use in defining complex XML vocabularies.

It seems many vendors amd individuals are realizing that the way to produce an innovative technology is for the vendors that will mostly be affected by the technology to come up with a specification that is satisfactory to the participants as opposed to trying to innovate by committee.  This is exactly what is happening with the next generation of the dynamic Web with the WHAT working group, with XML Web Services with WS-I and in syndication with RSS & Atom.

The W3C still has a good brand name since many associate it with the success of the Web but it seems that it has become damage that vendors route around in their bid to create the next generation of the World Wide Web.

 

At Microsoft one of our goals in developing software is that backwards compatibility when moving from one version of software to the next is high priority. However in certain cases the old behavior may be undesirable enough that we break compatibility. An example of such undesirable behavior are bugs that lead to incorrect results or security issues. Below is a list of breaking changes in the System.Xml namespace in beta 1 of v2.0 of the .NET Framework.

1. Extension of xs:anyType which changes the content type to mixed="false" results in error.
2. Add an enumeration member to XmlWriter.WriteState to indicate that the writer is in error state. Change writer to disallow further writes when in error state.
3. ##other namespace constraint now treated correctly on wild cards.
4. Instances of DateTime object returned by XmlValidatingReader that represent xs:time and other date & time related W3C XML Schema types now initialized using DateTime.MinValue
5. Incorrect implementation of XSD derivation hierarchy for xs:ENTITY and xs:NCName corrected.
6. XSD List Types Not Validated Correctly.
7. Changed to reliably fail when XmlTextReader source stream switches encoding between calls to ResetState()
8. XmlTextReader should apply the same security restrictions as the XmlReaders that can be created via the static XmlReader.Create() methods

I was directly involved in the decision making process for most of these breaking changes since many are in the W3C XML Schema area which I am responsible for. If any further clarifications about any of the breaking changes is needed, please post a comment with your question below.

 

I'm in the end stages of doing the spec work for the various components in the System.Xml namespace  I am responsible for in the Whidbey betas. After the 4th of July holidays we plan to start doing initial brain storming for what feature work we should do in Orcas/Longhorn. I thought it would be valuable to have various users of XML in the .NET Framework suggest what they'd like us to do in the Orcas version of System.Xml. What changes would people like to see? For example, I'm putting Schematron and XPathReader on the 'nice to have' list. No idea is too unconventional since this is the early brainstorming and prototyping phase.

Caveat: The fact that a technology is mentioned as being on our 'nice to have' list or is suggested in a comment to this post is not an indication that it will be implemented in future versions of the .NET Framework.

 

The folks at MSDN Chats have organized an online chat session on C# and XML for next month. The participants on the Microsoft side should include myself, Mark Fussell, Erik Meijer, Neetu Rajpal and couple of folks from the C# team. If you'd like to talk to us about topics surrounding XML and C# then log on to the XML and C# chat session at the following time

July 8, 2004
1:00 - 2:00 P.M. Pacific time
4:00 - 5:00 P.M. Eastern time
20:00 - 21:00 GMT

Event Reminders
Add to Outlook Calendar
Tell a Friend

On a side note, am I the only one that thinks the MSDN Chats site is crying out for an RSS feed? I definitely would love to add it to the subscriptions list in my favorite news aggregator.

 

My submission on Designing Extensible & Version Resilient XML Formats has been accepted to XML 2004. It looks like I'm going to be in Washington D.C. this fall. Currently I'm in the process of writing an article about the topic of my talk which should show up on MSDN and XML.com in the next month or so. Afterwards I plan to submit a revised version of that article as the paper for my talk.

 

The ongoing conversation between Jeremy Mazner and Jon Udell about the capabilities of WinFS deepen this morning with Jeremy's post Did I misunderstand Udell's argument against WinFS? which was followed up by Jon's post When a journalist blogs. In his post Jon asks

We have standard query languages (XPath, XQuery), and standard ways of writing schemas (XSD, Relax), and applications (Office 2003) that with herculean effort have been adapted to work with these query and schema languages, and free-text search further enhancing all this goodness. Strategically, why not build directly on top of these foundations?

Tactically, why do I want to write code like this:

public class Person
  {
  [XmlAttribute()] public string Title;
  [XmlAttribute()] public string FirstName;
  [XmlAttribute()] public string MiddleName;
  [XmlAttribute()] public string LastName;
  ....

in order to consume data like this?

<People>
  <Person
    DisplayName="Woodgrove Bank"
    IMAddress="Support@woodgrovebank.com"
    UserTile=".\user_tiles\Adventure Works.jpg">
    <EmailAddresses>
        <EmailAddress
            Type="Work"
            Address="mortgage@woodgrovebank.com"/>
        <EmailAddress
            Type="Primary"
            Address="Support@woodgrovebank.com"/>
   </EmailAddresses>

I believe two things to be true. First, we have some great XML-oriented data management technologies. Second, the ambitious goals of WinFS cannot be met solely with those technologies. I'm trying to spell out where the line is being drawn between interop and functionality, and why, and what that will mean for users, developers, and enterprises.

Jon asks several questions and I'll try to answer all the ones I can. The first question about why WinFS doesn't build on XML, XQuery and XSD instead of items, OPath and the WinFS schema language is something that the WinFS folks will have to answer. Of course, Jon could also ask why it doesn't build on RDF, RDQL [or any of the other RDF query languages] and RDF Schema which is a related question that naturally follows from the answer to Jon's question.

The second question is why would one want to program against a Person object when they have a <Person> element. This is question has an easy answer which unfortunately doesn't sit well with me. The fact is that developers prefer programming against objects than they do programming with XML APIs. No XML API in the .NET Framework (XmlReader, XPathNavigator, XmlDocument, etc) comes close to the ease of use of  programming against strongly typed objects in the general case. Addressing this failing [and it is a failing] is directly my responsibility since I'm responsible for core XML APIs in the .NET Framework. Coincidentally, we just had a review with our new general manager yesterday and this same issue came up and he asked what we plan to do about this in future releases. I have some ideas. The main problem with using objects to program against XML is that although objects work well for programming against data-centric XML (rigidly structured tabular data such as an the data in an Excel spreadsheet, a database dump or serialized objects) there is a signficant impedance mismatch when trying to use strongly typed objects to program against document-centric XML (semi-structured data such as a Word document). However the primary scenarios the WinFS folks want to tackle are about rigidly structured data which works fine with using objects as the primary programming model.

Jon says that he is trying to draw the line between interop and functionality. I'm curious as to what he means by interop in this case. The fact that WinFS is based on items, OPath and WinFS schema doesn't mean that WinFS data cannot be exchanged in an interoperable manner (e.g. some form of XML export and import) nor does it mean that non-Microsoft applications cannot interact with WinFS. I should clarify that I have no idea what the WinFS folks consider their primary interop scenarios but I don't think the way WinFS is designed today means it cannot interoperate with other platforms or data models.

I suspect that Jon doesn't really mean interop when says so. I believe he is using the word the same way Java people use it where it really means 'One Language, One Programming Model, One Platform' everywhere instead of being able to communicate between disparate end points. In this case the language is XML and the platform is the XML family of technologies.

 

In a post entitled My comments on the Infoworld article "Databases flex their XML" Michael Rys writes

Sean McCown wrote this analysis (PDF version) in Apr 2004. In the article, he compares the XML capabilities of the 4 major relational database systems (comparing publicly available versions) both in terms of functionality, ease, flexibility and speed, and adds a sidebar on Yukon. Before I start giving my comments on the article, let me disclose that I talked to Sean during his research for the article and answered his questions on SQL Server 2000 and Yukon. Thus, some of the comments below are just my attempts to make Sean's translation of my answers clearer, because I was not answering his questions clear enough .

Michael then goes on to clarify various points around the terminology used in the article, XQuery and SQL Server. Both Sean's article and Michael's followup are excellent reading for anyone interested in the growing trend of XML-enabled relational databases and how the big 3 relational database vendors stack up.

 

In a recent post entitled 15 Science Street Tim Bray, one of the inventors of XML, writes

Microsoft’s main talking point (I’m guessing here from the public documents) was that their software and format had the advantage that in WordML you can edit documents from arbitrary schemas.

Our pushback on that was that editing arbitrary-schema documents is damn hard and damn expensive and has never been anything more than a niche business.

which seems not to jibe with my experiences. Many businesses have XML formats specific to their target industry (LegalXML, HR-XML, FpML, etc) and many businesses use office productivity suites to create and edit documents. It seems very logical to expect that people would like to use their existing spreadsheet and word processing applications to edit their business documents instead of using XMl editors or specialized tools. More interestingly Tim Bray contradicts his position that editing user-defined schemas is a niche scenario when he writes

As we were winding up, a couple of really smart people (don’t know who they were) put up their hands and asked real good questions. The best was essentially “What would you like to see happen?” After some back and forth, I ended up with “You should have the right to own your own information. It’s your intellectual capital and you worked hard to produce it for your citizens. Sun doesn’t own it, Microsoft doesn’t own it, you own it, and that means it should be living in a nice, long-lived, non-proprietary data format that isn’t anyone’s competitive weapon.”

He took the words right out of my mouth. This is exactly what Microsoft has done with Office 2003 by allowing users to edit documents in XML formats of their choosing. In the letter Bringing the XML Vision to the Desktop with Office 2003 written by Jean Paoli of Microsoft (also a co-inventor of XML) he writes

an even greater and more innovative benefit is the fact that companies can now create their own XML schemas specific to their business, define the structure and type of data that each data element in a document contains and exchange information with customers and business partners more easily. This capability opens up a whole new realm of possibilities, not only for end users, but also for the business itself because now organizations can capture and reuse critical information that in the past has been lost or gone unused. 

Office 2003 is a great step forward in enabling businesses and end users harness the power of XML in typical document interchange scenarios. Arguments about whether you should use Sun's XML format or Microsoft's XML format aren't the point. The point is which tools allow you to use your XML format with the most ease.

 

 

 

Chris Sells has announced the call for speakers for the Applied XML Developers Conference 5. From his post

Are you interested in presenting a 45-minute talk on some applied XML or Web Services topic? It doesn't matter which platform or OS you're targeting. It also doesn't matter whether you're an author or vendor or professional speaker or a developer in the trenches (in fact, I tend to be biased towards the latter). We're after interesting and unique applications of XML and Web Services technology and if you're doing good work in that area, then I need you to send me a session topic and 2-4 sentence abstract along with a little bit about yourself. I'll be taking submissions 'til the end of June, but don't delay...

...the conference itself is likely to be in Oregon during the 2nd or 3rd week of September, 2004, but we're still working the details out. One of the fun things that we're thinking about this year is to have the Dev.Conf. in Sunriver, Oregon, a resort and spa town in central Oregon where sun is plentiful and rain is scarce.

Previous XML DevCons have had a wide variety of interesting speakers. Unfortunately, the XML DevCon webpage doesn't provide any information on previous conferences. If you are interested in reports on last year's conference just type "XML DevCon" in your favorite Web search engine to locate blog postings from some of the attendees.

I probably won't be at this conference since the focus is usually XML Web Services while my professional interests are in core XML technologies with working with XML syndication formats being a hobby. However there should be lots of interesting presentations on XML Web Services and other leading edge applications of XML from industry experts if last year's conference is anything to go by.

 

Jon Udell has started a series of blog posts about the pillars of Longhorn.  So far he has written Questions about Longhorn, part 1: WinFS and Questions about Longhorn, part 2: WinFS and semantics which ask the key question "If the software industry and significant parts of Microsoft such as Office and Indigo have decided on XML as the data interchange format, why is the next generation file system for Windows basically an object oriented database instead of an XML-centric database?" 

I'd be very interested in what the WinFS folks like Mike Deem would say in response to Jon if they read his blog. Personally, I worry less about how well WinFS supports XML and more about whether it will be fast, secure and failure resistant. After all, at worst WinFS will support XML as well as a regular file system does today which is good enough for me to locate and query documents with my favorite XML query language today. On the other hand, if WinFS doesn't perform well or shows the same good-idea-but-poorly-implemented nature of the Windows registry then it'll be a non-starter or much worse a widely used but often cursed aspect of Windows development (just like the Windows registry).

As Jon Udell points out the core scenarios touted for the encouraging the creation of WinFS (i.e search and adding metadata to files) don't really need a solution as complex or as intrusive to the operating system as WinFS. The only justification for something as radical and complex as WinFS is if Windows application developers end up utilizing it to meet their needs. However as an application developer on the Windows platform I primarily worry about three major aspects of WinFS. The first is performance, I definitely think having a query language over an optimized store in the file system is all good but wouldn't use it if the performance wasn't up to snuff. Secondly I worry about security, Longhorn evangelists like talking up what a wonderful world it would be if all my apps could share their data but ignore the fact that in reality this can lead to disasters. Having multiple applications share the same data store where one badly written application can corrupt the entire store is worrisome. This is the fundamental problem with the Windows registry and to a lesser extent the cause of DLL hell in Windows. The third thing I worry about is that the programming model will suck. An easy to use programming model often trumps almost any problem. Developers prefer building distributed applications using XML Web Services in .NET to the alternatives even though in some cases this choice leads to lower performance. The same developers would rather store information in the registry than come up with a robust alternative on their own because the programming model for the registry is fairly straightforward.

All things said, I think WinFS is an interesting idea. I'm still not sure it is a good idea but it is definitely interesting. Then again given that WinFS assimilated and thus delayed a very good idea from shipping, I may just be a biased SOB.

PS: I just saw that Jeremy Mazner posted a followup to Jon Udell's post entitled Jon Udell questions the value and direction of WinFS where he wrote

XML formats with well-defined, licensed schemas, are certainly a great step towards a world of open data interchange.  But XML files alone don't make it easier for users to find, relate and act on their information. Jon's contention is that full text search over XML files is good enough, but is it really?  I did a series of blog entries on WinFS scenarios back in February, and I don't think's Jon full text search approach would really enable these things. 

Jeremy mostly misses Jon's point which is aptly reduced to a single question at the beginning of this post. Jon isn't comparing full text search over random XML files on your file system to WinFS. He is asking why couldn't WinFS be based on XML instead of being an object oriented database.

 

One of my friends, Joshua Allen, is a fan of RDF and Semantic Web technologies. Given that I respect his opinion a lot I keep trying to delve into RDF and its family of technologies every couple of months to see what it provides to the world of data access and information interchange above and beyond existing technologies. Recently I discovered that there are some in the RDF camp that position it as a "better XML". The first example of this I saw was an old article by Tim Berners-Lee entitled Why RDF model is different from the XML model. According to Tim the note is an attempt to answer the question, "Why should I use RDF - why not just XML?". However instead of answering the question his note just left me with more questions than answers. The pivotal point for me in Tim Berners-Lee's note is the following excerpt

Things you can do with RDF which you can't do with XML include

• You can parse the semantic tree, which end up giving you a set of (possibly mutually referential) triples and then you can use the ones you want ignoring the ones you don't understand.

Problems with basing you understanding on the structure include

• Without having gone to the trouble of getting the schema, or having an application hand-programmed to recognise a particular document type, you can't pick up any semantic information from a document;
• When an XML schema changes, it could typically introduce new intermediate elements (like "details" in the tree above or "div" is HTML). These may or may or may not invalidate any query which has been based on the structure of the document.
• If you haven't gone to the trouble of making a semantic model, then you may not have a well defined one.

It seems that the point being argued is that with RDF you can get more understanding of the information in the document than with just XML. Being that one could consider RDF as just a logical model layered on top of an XML document (e.g. RDF/XML) I find it hard to understand how viewing some XML document through RDF colored glasses buys one so much more understanding of the data.

Recently I discovered a presentation entitled REST, Self-description, and XML by Mark Baker. This presentation discusses the ideas in Tim Berners-Lee's note in more depth and in a way I finally understand. The first key idea in Mark's presentation is the notion of "self describing" data formats which were also covered in Tim Berners-Lee's presentation at WWW2002 entitled Specs Count. The core tennets of "self describing" data formats are covered in slide 10 and slide 11 of Mark's presentation. A "self describing" data formats contains all the data needed to figure out how to process the format from publically accessible specs. For example, an HTTP response tells you the MIME type of the document which can be used to locate the appropriate RFC which governs how the format should be processed. In the case of XML, Tim Berners-Lee states that an HTTP response which returns an XML document either as application\xml or text\xml should be processed according to the rules of the XML and XML namespaces recommendations which state that the identity of an element is determined based on its namespace name. So when processing an XML document, Tim asserts that it is self describing because one can locate the spec for the format from the namespace URI of the root element. Of course, Mark disagrees with this but his reasons for doing so is pedantic spec lawyering. I disagree with it as well but for different reasons. The main reason I disagree with it is because it puts a stake in the ground and says that any XML format on the Web that doesn't use namespace name for its root element or whose namespace name is not a dereferenceable URI that leads to a spec is broken. This automatically states that XML formats used on the Web today such as RSS 1.0, RSS 2.0, OPML and the Atom 0.3 syndication format are broken.

Mark then goes on to state in slide 20 that a problem with XML formats is that one can't arbitrarily extend an XML document without it's schema or without breaking some application somewhere. It's unclear as to what he means by the document's schema but will grant that it is likely that arbitrary additions to the expected content of an XML document will break certain applications. Getting to slide 24, it is slightly clearer what Mark is getting at. He claims that one although one can add extend a format by adding extra elements from a known namespace using just XML technologies this doesn't tell you how to deal with the extensions. On the other hand, with RDF the extensions are all concepts named with a URI whose meaning can then be looked up using HTTP GET. This is where he lost me. I don't see the difference between seeing a namespaced XML element in an XML format and using HTTP GET on the namespace URI of the element to locate the spec or schema for the namespaced extension and what he describes as the gains of using RDF.

The more I look at how RDF people bag on XML the more it seems that they don't really write applications in today's world. Almost every situation I've seen someone claim that RDF technologies will in the future be able to solve a problem XML cannot, the problem is actually not only solveable with XML technologies but actually is being solved using XML technologies today.  

 

As I mentioned yesterday Doug Purdy posted an insightful entry in response to Ted Neward's about the inappropriateness of returning ADO.NET DataSets from XML Web Services. Today Ted Neward has a post entitled  Why Purchase Orders are the root of all evil? which almost entirely misses the point of Doug's post.

Ted writes

Could you tell me what the schema should be? Doug, it's right there in front of you: the class definition itself is a schema definition for objects of its type. The question I think you mean to ask is, "What the XML schema should be for this Purchase Order?", but I can't do that, because you've already stepped way out into la-la land as far as XML/XSD goes by making use of generic types (like Dictionary) for which there is no XSD equivalent; sure, we can rpc-encode one up, but we're back to turning objects into XML and back again, and I thought we didn't like that....?

Could you tell me what each particle of the schema means? Well, the LineItemAddedEvent certainly isn't a schema construct, so I'm guessing that'll have to be the XML-based representation of a .NET delegate.... the IAddress has no implementation behind it that I can see so once again I'll have to punt....

Oh, I get it... Doug's using one of them anti-pattern thingies to show us what not to do when trying to define types in XML/XSD for use in Web services (or WebServices or web services or however we've decided to spell these silly things anyway).

You're absolutely right, Doug--the way that thing is written, Purchase Orders, while perhaps not the root of ALL evil, are certainly evil and therefore should be banned from the WS-* camp immediately.

Seriously, dude, DataSets as return values from Web services are evil. Get over it.

What I find interesting is that Ted Neward is looking at XML Web Services through the perspective of distributed objects. His entire arguments hinge around the fact that his applications convert XML into Java or CLR objects so the XML returned must be something that is condusive to converting to objects easily. Doug accurately points out that there is no one-to-one mapping between an XML schema and a CLR object. Arguing that your favorite platform has one-to-one mappings for some XML schemas and not others thus banning various XML formats from participating in XML Web Services is a very limiting viewpoint. I'd like to ask Ted whether he also would ban XBRL, wordProcessingML or UBL documents from being used in XML Web Services because there aren't easy ways to convert them to a handy, dandy Java object with strongly typed members and all that jazz.  

I don't dispute the practical reasons for discouraging developers from returning ADO.NET DataSets from XML Web Services since most developers trying to access the XML Web Services just use a toolkit that pretends you are building distributed object applications. Usually such toolkits either barf horribly when faced with XML they don't grok or force developers to have deal with scary angle brackets directly instead of the object facade they know & love (ASP.NET XML Web Services included). This is a practical reason to avoid exposing ADO.NET DataSets from XML Web Services that may be accessed from Java platforms especially since such platforms don't make it easy to deal with raw XML.

On the other hand, claiming that there is some philosophical reason not to expose data from an XML Web Service that may be be semi-structured and full of unknown data (i.e XML data) seems quite antithetical to the entire point of XML Web Services and the Service Oriented Architecture fad.

 

 

It seems every few months there are a series of blog posts or articles about why returning ADO.NET DataSet objects from XML Web Services.  I saw the most recent incarnation of this perma-debate in Scott Hansellman's post Returning DataSets from WebServices is the Spawn of Satan and Represents All That Is Truly Evil in the World and Ted Neward's More on why DataSets are the Root of all Evil.

I was going to type up a response to both posts until I saw Doug Purdy's amusing response, PurchaseOrders are the root of all evil, which succintly points out the flaws in Scott and Ted's arguments.

Now I'm off to bed.

 

I just read Tim Bray's entry entitled SOA Talk where he mentions listening to Steve Gillmor, Doc Searls, Jon Udell, Dana Gardner, and Dan Farber talk about SOA via “The Gillmor Gang” at ITConversations. I tried to listen to the radio show a few days ago but had the same problems Tim had. A transcript would definitely be appreciated.

What I found interesting is this excerpt from Tim Bray's blog post

Apparently a recent large-scale survey of professionals revealed that “SOA” has positive buzz and high perceived relevance, while “Web Services” scores very low. Huh?

This is very unsurprising to me. Regular readers of my blog may remember I wrote about the rise of the Service Oriented Architecture fad a few months ago. Based on various conversations with different people involved with XML Web Services and SOA I tend to think my initial observations in that post were accurate. Specifically I wrote

The way I see it the phrase "XML Web Services" already had the baggage of WSDL, SOAP, UDDI, et al so there a new buzzphrase was needed that highlighted the useful aspects of "XML Web Services" but didn't tie people to one implementation of these ideas but also adopted the stance that approaches such as CORBA or REST make sense as well.

Of the three words in the phrase "XML Web Services" the first two are implementation specific and not in a good way. XML is good thing primarily because it is supported by lots of platforms and lots of vendors not because of any inherrent suitability of the technology for a number of the tasks people utilize it for. However in situations where this interop is not really necessary then XML is not really a good idea. In the past, various distributed computing afficionados have tried to get around this by talking up the The InfoSet which was just a nice way of deprecating the notion of usage of the XML text format everywhere being a good thing. The second word in the phrase is similarly inapllicable in the general case. Most of the people interested in XML Web Services are interested in distributed computing which traditionally and currently is more about the intranet than it is about the internet. The need to justify the Web-like nature of XML Web Services when in truth these technologies probably aren't going to be embraced on the Web in a big way seems to have been a sore point of many discussions in distributed computing circles.

Another reason I see for XML Web Services having negative buzz versus SOA is that when many people think of XML Web Services, they think of overhyped technologies that never delivered such as Microsoft's Hailstorm.  On the other hand, SOA is about applying the experiences of 2 decades of building distributed applications to building such applications today and in the future. Of course, there are folks at Microsoft who are wary of being burned by the hype bandwagon and there've already been some moves by some of the thought leadership to distance what Microsoft is doing from the SOA hype. One example of this is the observation that lots of the Indigo folks now talk about 'Service Orientation' instead of 'Service Oriented Architecture'.

Disclaimer: The above comments do not represent the thoughts, intentions, plans or strategies of my employer. They are solely my opinion.

 

C.J. Date, one of the most influential names in the relational database world, had some harsh words about XML's encroachment into the world of relational databases in a recent article entitled Date defends relational model  that appeared on SearchDatabases.com. Key parts of the article are excerpted below

Date reserved his harshest criticism for the competition, namely object-oriented and XML-based DBMSs. Calling them "the latest fashions in the computer world," Date said he rejects the argument that relational DBMSs are yesterday's news. Fans of object-oriented database systems "see flaws in the relational model because they don't fully understand it," he said.

Date also said that XML enthusiasts have gone overboard.

"XML was invented to solve the problem of data interchange, but having solved that, they now want to take over the world," he said. "With XML, it's like we forget what we are supposed to be doing, and focus instead on how to do it."

Craig S. Mullins, the director of technology planning at BMC Software and a SearchDatabase.com expert, shares Date's opinion of XML. It can be worthwhile, Mullins said, as long as XML is only used as a method of taking data and putting it into a DBMS. But Mullins cautioned that XML data that is stored in relational DBMSs as whole documents will be useless if the data needs to be queried, and he stressed Date's point that XML is not a real data model.

Craig Mullins points are more straightforward to answer since his comments don't jibe with the current state of the art in the XML world. He states that you can't query XML documents stored in databases but this is untrue. Almost three years ago, I was writing articles about querying XML documents stored in relational databases. Storing XML in a relational database doesn't mean it has to be stored in as an opaque binary BLOB or as a big, bunch of text which cannot effectively be queried. The next version of SQL Server will have extensive capabilities for querying XML data in relational database and doing joins across relational and XML data, a lot of this functionality is described in the article on XML Support in SQL Server 2005. As for XML not having a data model, I beg to differ. There is a data model for XML that many applications and people adhere to, often without realizing that they are doing so. This data model is the XPath 1.0 data model, which is being updated to handled typed data as the XQuery and XPath 2.0 data model.

Now to tackle the meat of C.J. Date's criticisms which is that XML solves the problem of data interchange but now is showing up in the database. The thing first point I'd like point out is that there are two broad usage patterns of XML, it  is used to represent both rigidly structured tabular data (e.g., relational data or serialized objects) and semi-structured data (e.g., office documents). The latter type of data will only grow now that office productivity software like Microsoft Office have enabled users to save their documents as XML instead of proprietary binary formats. In many cases, these documents cannot simply shredded into relational tables. Sure you can shred an Excel spreadsheet written in spreadsheetML into relational tables but is the same really feasible for a Word document written in WordprocessingML? Many enterprises would rather have their important business data being stored and queried from a unified location instead of the current situation where some data is in document management systems, some hangs around as random files in people's folders while some sits in a database management system.

As for stating that critics of the relational model don't understand it, I disagree. One of the major benefits of using XML in relational databases is that it is a lot easier to deal with fluid schemas or data with sparse entries with XML. When the shape of the data tends to change or is not fixed the relational model is simply not designed to deal with this. Constantly changing your database schema is simply not feasible and there is no easy way to provide the extensibility of XML where one can say "after the X element, any element from any namespace can appear". How would one describe the capacity to store “any data” in a traditional relational database without resorting to an opaque blob?

I do tend to agree that some people are going overboard and trying to model their data hierarchically instead of relationally which experience has thought us is a bad idea. Recently on the XML-DEV mailing list entitled Designing XML to Support Information Evolution where Roger L. Costello described his travails trying to model his data which was being transferred as XML in a hierarchical manner. Micheal Champion accurately described the process Roger Costello went through as having "rediscovered the relational model". In a response to that thread I wrote "Hierarchical databases failed for a reason".

Using hierarchy as a primary way to model data is bad for at least the following reasons

1. Hierarchies tend to encourage redundancy. Imagine I have a <Customer> element who has one or more <ShippingAddress> elements as children as well as one or more <Order> elements as children as well. Each order was shipped to an address, so if modelled hierarchically each <Order> element also will have a <ShippingAddress> element which leads to a lot of unnecessary duplication of data.
2. In the real world, there are often multiple groups to which a piece of data belongs which often cannot be modelled with a single hierarchy.  
3. Data is too tightly coupled. If I delete a <Customer> element, this means I've automatically deleted his entire order history since all the <Order> elements are children of <Customer>. Similarly if I query for a <Customer>, I end up getting all the <Order> information as well.

To put it simply, experience has taught the software world that the relational model is a better way to model data than the hierarchical model. Unfortunately, in the rush to embrace XML many a repreating the mistakes from decades ago in the new millenium.

 

XML.com recently ran an article entitled Document-Centric .NET, that highlights the various technologies for working with XML that exist in the .NET Framework. The article provides a good high level overview of the various options you have for processing XML in the .NET Framework. The article includes an all important caveat which I wish more people knew about and which I keep wanting to write an article about but never get around to doing. The author writes 

However, keep in mind that there are W3C XML Schema features that are not directly compatible with .NET's XML-to-database and XML-to-object mapping tools.

This is very true. Besides our schema validation technologies, most Microsoft technologies or products that utilize W3C XML Schema support a subset of the language due to impedance mismatches between the language and the underlying data model or type system of the target environment.

In fact the only complaint I have about the article is a nitpick about its title. In XML circles, document-centric implies a usage of XML that isn't borne out by his article. If you are interested in the difference between data-centric XML and document-centric XML you should read my article Can One Size Fit All? in XML Journal. In that article I talk about the differences between XML that is used to represent both rigidly structured tabular data (e.g., relational data or serialized objects) and semi-structured data (e.g., office documents). The former is data-centric XML while the latter is document-centric.

 

 

I recently stumbled on an entry by Lucas Gonze where he complains about the RSS <enclosure> element. He writes

Problems with the enclosure element:

• It causes users to download big files that they will never listen to or watch, creating pointless overload on web hosts.
• It doesn't allow us to credit the MP3 host, so we can't satisfy the netiquette of always linking back.
• For broadband users, MP3s are not big enough to need advance caching in the first place.
• The required content-type attribute is a bad idea in the first place. Mime settings are already prone to breakage, adding an intermediary will just create another source of bugs. There are no usecases for this attribute that can't be more easily and robustly satisfied by having clients HEAD the URL for themselves.
• The required content-length attribute should not be there. It requires people who link to MP3s to HEAD them and calculate the length, which is sometimes not practical. It makes variable-length MP3s illegal. There are no usecases for this attribute that can't be more easily and robustly satisfied by having clients HEAD the URL for themselves.

The primary problem with the <enclosure> element is that it is overspecified. Having an element that says, here is a pointer to some data that is related to this entry that is too large to fit in the feed is a good idea. Similarly providing a hint at what the MIME type is so the reader knows whether it can handle that MIME type or can display something specific to that media type in the user interface without making an additional request to the server is very useful. The description of the enclosure element in RSS 2.0 states

<enclosure> sub-element of <item> 

<enclosure> is an optional sub-element of <item>.

It has three required attributes. url says where the enclosure is located, length says how big it is in bytes, and type says what its type is, a standard MIME type.

The url must be an http url.

<enclosure url="http://www.scripting.com/mp3s/weatherReportSuite.mp3" length="12216320" type="audio/mpeg" />

Syndication geeks might notice that this is akin to the <link> element in the ATOM 0.3 syndication format which is described as

3.4  Link Constructs

A Link construct is an element that MUST NOT have any child content, and has the following attributes:

3.4.1  "rel" Attribute

The "rel" attribute indicates the type of relationship that the link represents. Link constructs MUST have a rel attribute, whose value MUST be a string, and MUST be one of the values enumerated in the Atom API specification <eref>http://bitworking.org/projects/atom/draft-gregorio-09.html</eref>.

3.4.2  "type" Attribute

The "type" attribute indicates an advisory media type; it MAY be used as a hint to determine the type of the representation which should be returned when the URI in the href attribute is dereferenced. Note that the type attribute does not override the actual media type returned with the representation.

Link constructs MUST have a type attribute, whose value MUST be a registered media type [RFC2045].

3.4.3  "href" Attribute

The "href" attribute contains the link's URI. Link constructs MUST have a href attribute, whose value MUST be a URI [RFC2396].

xml:base [W3C.REC-xmlbase-20010627] processing MUST be applied to the atom:url element.

3.4.4  "title" Attribute

The "title" attribute conveys human-readable information about the link. Link constructs MAY have a title attribute, whose value MUST be a string.

So the ideas behind the <enclosure> element were good enough that they appear in ATOM with some additional niceties and a troublesome bit (the length attribute) removed. So if the concepts behid the <enclosure> element are so good that they are first class members of the ATOM syndication format. Why does Lucas not like it? The big problem with RSS enclosures is how Dave Winer expected them to be used. An aggregator was supposed to act like a TiVo, automatically downloading files in the background and presenting them to you when it's done. The glaring problem with doing this is that it means lots of people are automatically downloading large files that they didn't request which is a significant waste of bandwidth. In fact, most aggregators either do not support enclosures or simply show them as links which is what FeedDemon and RSS Bandit (with the Outlook 2K3 skin) do. The funny thing is that the actual RSS specification doesn't describe this behavior, instead this behavior is implied by Dave Winer's descriptions of use cases.

Lucas also complains about the required length attribute which is problematic if you are pointing to a file on a server you don't own because you have to first download the file or perform a HTTP HEAD to get its size. The average blogger isn't going to go through that kind of trouble. Although tools could help it makes sense for the  length attribute  to have been an optional hint.

I have to disagree with Lucas's complaints about putting the MIME type in the <enclosure> element. He complains that the MIME type in the <enclosure> could be wrong and in fact that in many cases web servers  serve a file with the wrong MIME type. Thus he concludes that putting the MIME type in the enclosure is wrong. Client software should be able, to decide how to react to the enclosure [e.g. if it is audio/mpeg display a play button] without having to make additional HTTP requests especially since as Lucas points out it is not a 100% guaranteed that performing an HTTP HEAD of the linked file will actually get you the correct MIME type from the web server.

In conclusion, I agree that the <enclosure> element is problematic but most of the problems are due to the implied use case suggested by the spec author, Dave Winer, as opposed to the actual information provided by the element. The ATOM approach of describing the information provided by each element in a feed but not explicitly describing the expected behavior of clients is a welcome approach. Of course, there will always be developers who require structure or take an absence of explicit guidelines to mean do stupid things (like aggregators that fetch your feed every 5 minutes)  but these are probably better handled in "Best Practices" style documents or test suites than in the actual specification.

 

One of the hardest problems in software development is how to version software and data formats. One of the biggest problems for Windows for years has been DLL Hell which is a versioning problem. One of the big issues I have to deal with at work is how to deal with versioning issues when adding or removing functionality from classes.

For a few weeks, I've been planning to write up some guidelines and concerns for versioning XML formats based on my experiences and those of others at Microsoft. I've got some folks on the XML Web Services team interested in riding shotgun such as Gudge and Doug. It also looks like Edd Dumbill is interested in the abstract for the article, so it with any luck it should end up on XML.com when it is done.

I was reminded of the importance of writing this article when I saw a post on the atom-syntax list by Google's Steve Jensen which implied that it just occured to the folks at Google that they'd have to support multiple versions of ATOM. This is excarberated by the fact that they are deploying implementations based on draft specs. Like I said before, never ascribe to malice that which can be explained by incompetence

 

I've mentioned in the past why I think XML 1.1 was a bad idea in my post XML 1.1: The W3C Gets It Wrong. It seems at least one W3C working group, the XML Protocols working group to be exact, has now realized why XML 1.1 is a bad idea a few months later. Mark Nottingham recently posted a message to the W3C Technical Architecture Group's mailing list entitled Deployment and Support of XML 1.1 where he writes

In the Working Group's view, this highlights a growing misalignment in
the XML architecture. Until the advent of XML 1.1, XML 1.0 was a single
point of constraint in the XML stack, with all of the benefits (e.g.,
interoperability, simplicity) that implies. Because XML 1.1 has
introduced variability where before there was only one choice, other
standards now need to explicitly identify what versions of XML they are
compatible with. This may lead to a chicken-and-egg problem; until
there is a complete stack of XML 1.1-capable standards available, it is
problematic to use it.

Furthermore, XML-based applications will likewise need to identify
their capabilities and constraints; unfortunately, there is no
consistent way to do this in the Web architecture (e.g., RFC3023 does
not provide a means of specifying XML versions in media types).

As I mentioned in my previous post about the topic, XML 1.1 hurts the interoperability story of XML which is one of the major reasons of using it in the first place. Unfortunately, the cat is already out of the bag, all we can do now is try to contain or avoid it without getting our eyes clawed out. I tend to agree with my coworker Michael Rys, the day XML 1.1 became a W3C recommendation was a day of mourning.

 

The next version of SQL Server will have a significant amount of functionality related to storing, querying and extracting XML from the database. To accompany the information being imparted at TechEd 2004, myself and rest of the folks behind the XML Developer Center on MSDN decided to run a series of articles on the XML features of SQL Server 2005. The articles will run through the month of June.

The first article in the series is XML Support in Microsoft SQL Server 2005. Read this article if you are interested in learning how SQL Server 2005 has become an fully XML-aware database including the addition of the XML datatype, support for XML Schemas, indexing  of XML data, XQuery, querying XML views of relational data using XPath and much more.

 

Joshua Allen has a post entitled RSS Politics which does a good job of properly framing the growing Microsoft and RSS vs. Google and Atom silliness spurred by Joi Ito that I've been seeing in the comments on Robert Scoble's weblog. Joshua writes

First, be very clear.  The “debate“ over Atom vs. RSS is a complete non-issue for Microsoft.  We use RSS to serve thousands of customers right now, and most of the people setting up RSS feeds have never heard of the political “debates“.  RSS works for them, and that's all they care about.  On the other hand, if Atom ever reaches v1.0 and we had a business incentive to use it, we would use it.  No need for debate.

Now, of the three or four people at Microsoft who know enough about Atom to have said anything about it, I wouldn't say that anyone has trashed the format.  I and others have pointed out that it's just fine for what it does; just like RSS.  If anything, I have asked hard questions about why I or any business decision maker should be spending resources on the whole debate right now.  If a business has deployed using RSS, what financial motive would they have to switch to a new, nearly identical, format once it ships?  I've got nothing against the Atom people inventing new syndication formats, but I just don't see why *I* should be involved right now.  There's no good reason.

The other comment I've made before is that the Atom community is not being served by the polarizing attitudes of some participants.  The “us vs. them“ comments are not helpful, especially when untrue, and the constant personalization (”Support Atom because I hate Dave Winer!”) just damages the credibility of the whole group (many of whom might have good motives for being involved).

I totally echo his sentiments. In the past couple of months more and more folks at Microsoft have pinged me about syndication and blogging technologies once they learn I wrote RSS Bandit. Every single time I've given them the same advice I gave in my post, Mr. Safe's Guide to the RSS vs. ATOM debate. If you are a feed consumer you'll need to support the various flavors of RSS and the various flavors of ATOM (of which there'll at least be two, ATOM 0.3 and whatever is produced from the IETF/W3C process). If you are a feed producer, you should stick with RSS 0.91/2.0 since this is the widest supported format and the most straightforward.

Although no one has asked yet, I'm also going to give my advice on Mr. Safe at Microsoft should consider adopting the ATOM API. In my personal opinion, the current draft of the ATOM API seems better designed and falls more inline with Microsoft's technologies than the existing alternatives (Blogger API/MetaWeblog API/LiveJournal API), etc. However the API lacks lots of functionality and in fact already there are extensions to the ATOM API showing up in the wild. Currently, these "innovations" are being lauded but given the personalities behind ATOM it is likely that if Microsoft products supported the API and extended it there could be a negative backlash. In which case perhaps going with a product specific API may be the best option if there is sensitivity to such feedback or the ATOM API has to be significantly extended to fit the product's needs.

 

I've posted a few entries in the past questioning the value of the Semantic Web as currently envisioned by the W3C along with its associated technologies like RDF and OWL. My most recent post about this was On Semantic Integration and XML. It seems I'm not the only XML geek who's been asking the same questions after taking a look at the Semantic Web landscape. Elliotte Rusty Harrold is at WWW2004 and wrote the following opinions of the Semantic Web on Day 4 of WWW2004

This conference is making me think a lot about the semantic web. I'm certainly learning more about the details (RDF, OWL etc.). However, I still don't see the point. For instance what does RDF bring to the party? The basic idea of RDF is that a collection of URIs forms a vocabulary. Different organizations and people define different vocabularies, and the URIs sort out whose name, date, title, etc. property you're using at any given time. Remind you of anything? It reminds me a lot of XML + namespaces. What exactly does RDF bring to the party? OWL (if I understand it) lets you connect different vocabularies. But so does XSLT. I guess the RDF model is a little simpler. It's all just triples, that can be automatically combined with other triples, and thereby inferences can be drawn. Does this actually produce anything useful, though? I don't see the killer app. Theoretically a lot of people are talking about combining RDF and ontologies from mulktiple sources too find knowledge that isn't obvious from any one source. However, no one's actually publishing their RDF. They're all transforming to HTML and publishing that.

I've written variations of the same theme over the past couple of months. It's just hard to point at any practical value that RDF/OWL/etc provide over XML/XSLT/etc for semantic integration.

 

I've been reading the various pieces of feedback on my recent blog post on Why You Won't See XSLT 2.0 or XPath 2.0 in the Next Version of the .NET Framework including the 40 comments in response to the post and the "Microsoft is killing XSLT" thread on xsl-list. Most of it has been flames witrh little useful feedback but there was an interesting response by Norm Walsh entitled XQuery 1.0 or XSLT 2.0? which I've been drawn to respond to. Norm writes

Dare Obasanjo argues that “XQuery is strongly and statically typed while XPath 2.0 is weakly and dynamically typed.” What’s not clear from his post is that he is comparing XQuery 1.0 to XPath 2.0 in backwards compatibility mode (Michael Rys did provide a clarification). That’s an odd comparison to make. XPath 2.0 needs a backwards compatibility mode so that it stands some chance of doing the right thing when used in the context of an XSLT 1.0 stylesheet, but that’s not the expected mode for long-term use.

I thought my point was self evident here but if Norm missed it then it means most of the people who read my original blog post did as well. XPath 2.0 is a subset of XQuery 1.0, the parts of XQuery missing are XML construction, the query prolog, the let-where-orderby parts of the FLWOR expression, typeswitch and a few other things.  XPath 2.0 has a backwards compatibility mode which has different semantics from regular XPath 2.0 and XQuery. When I talked about Microsoft not implementing XPath 2.0 I meant XPath 2.0 in backwards compatibility mode since implementing XQuery means you already have regular XPath 2.0. After all, everything you can do in XPath 2.0 you can do in XQuery. 

Norm also writes

The funniest arguments are the ones that imply that XQuery is a competitor in the same problem space as XSLT, that users will use XQuery instead of XSLT. I say that’s funny because there are so many problems that you simply cannot solve with XQuery. If your data is regular and especially if it’s all stored in a database already so that your XQuery implementation can run really fast, then XQuery absolutely makes sense, but didn’t the database folks already have a query language? Nevermind. If your customers don’t need to solve the kinds of problems for which XSLT was designed, or if you want to sell them some sort of proprietary system to solve them, then implementing XSLT 2.0 probably doesn’t make sense.

I've seen variations of the above theme (XSLT is for transformation, XQuery is for query) in various responses to my original post. Taking away the words query and transformation out of the picture both XQuery and XSLT are designed to reshape XML data. SQL is primarily a query language but you can use it to reshape relational data, this is exactly how SQL views work. For most people, the transformations they want to perform using XSLT also be expressed using XQuery. Per Bothner wrote an article over a year ago on XML.com about Generating XML and HTML using XQuery showing how you could use XQuery to transform an XML document to another XML format or HTML. There are a few niceties in XSLT 2.0 that don't exist in XQuery such as the ability to write to multiple output streams but in general most of the things you can do in XSLT 2.0 can also be done in XQuery. In fact this leads me to something else Norm wrote

If you want to transform documents that aren’t regular, especially documents that have a lot of mixed content, XSLT is clearly the right answer. I’ll wager dinner at your favorite restaurant that XQuery cannot be used to implement the functionality of the DocBook XSLT Stylesheets. (You produce the XQuery that does the job, I buy you dinner.)

First of all XSLT is actually very bad at dealing with XML that isn't regular and has lots of mixed content. This is why a number of XSLT gurus got together to created EXSLT and why I started the EXSLT.NET project (grab the latest version from the Microsoft.com download servers here). As for transforming DocBook with XQuery, as I mentioned before Per Bothner wrote an article about using XQuery for transformations. In fact, he specifically writes about Transforming DocBook to HTML using XQuery.

The bottom line is that XQuery is as much a "transformation language" as XSLT. XSLT may have some functionality that XQuery does not have but there isn't much I've seen that couldn't be implemented using extension functions. Perhaps I should start an EXQuery.NET project? :)

 

 

A few months ago Mark Fussell wrote an article entitled What's New in System.Xml for Visual Studio 2005 and the .NET Framework 2.0 Release. Mark Ihimoyan has a followup series of blog posts which mentions which of the new features of System.Xml mentioned in Mark's article will actually be in the .NET Compact Framework. The blog posts are listed below

1. System.Xml in NETCF v2.0 part I
2. System.Xml in NETCF v2.0 part II
3. System.Xml in NETCF v2.0 part III

 

 

The Microsoft Pattern and Practices folks have produced an excellent guide to Improving .NET Application Performance and Scalability with a chapter on Improving XML Performance. If you build .NET Framework applications that utilize XML then you owe it to yourself to take a look at the guidelines in that document. There is also a handy, easily printable XML Performance checklist which can be used as a quick way to check that your application is doing the right thing with regards to getting the best performance for XML applications.

On a similar note, Mark Fussell has posted XmlNameTable: The Shiftstick of System.Xml and XmlNameTable Revisited which provide some tips about how to use the XmlNameTable class to improve processing speed by up to 10% when processing XML documents.

 

Charles Cook has a blog posting on XML and Performance where he writes

XML-based Web Services look great in theory but I had one nagging thought last week while on the WSA course: what about performance? From my experience with VoiceXML over the last year it is obvious that processing XML can soak up a lot of CPU and I was therefore interested to see this blog post by Jon Udell in which he describes how Groove had problems with XML:

Sayonara, top-to-bottom XML I don't believe that I pay a performance penalty for using XML, and depending on how you use XML, you may not believe that you do either. But don't tell that to Jack Ozzie. The original architectural pillars of Groove were COM, for software extensibility, and XML, for data extensibility. In V3 the internal XML datastore switches over to a binary record-oriented database.

You can't argue with results: after beating his brains out for a couple of years, Jack can finally point to a noticeable speedup in an app that has historically struggled even on modern hardware. The downside? Debugging. It was great to be able to look at an internal Groove transaction and simply be able to read it, Jack says, and now he can't. Hey, you've got to break some eggs to make an omelette.

Is a binary representation of the XML Infoset a useful way of improving performance when handling XML? Would it make a big enough difference?

For the specific case of Groove I'd be surprised if they used a binary representation of the XML infoset as opposed to a binary representation of their application object model. Lots of applications that utilize XML for data storage or configuration data immediately populate this data into application objects. This is a layer of unnecessary processing since one could skip the XML reading and writing step and directly read and write serialized binary objects. If performance is that important to your application and there are no interoperability requirements it is a better choice to serialize binary objects instead of going through the overhead of XML serialization/deserialization. The main benefit of using XML in such scenarios is that in many cases there is existing infrastructure for working with XML such as parsers, XML serialization toolkits and configuration handlers. If your performance requirements are so high that the overhead of going from XML to application objects is too high then getting rid of the step in the middle is a wise decision. Although as pointed out by Jon Udell you loose the ease of debugging that comes with using a text based format.

If you are considering using XML in your applications always take the XML Litmus Test