I just found out that more than 55,000 people have entered the XML challenge so far. Remember that some contests in the challenge are run on a monthly basis, so there is still an opportunity to win prizes. And I understand that there are still more than $50,000 in prizes that have yet to be won in US and Canada. I also heard that the organizers still have t-shirts to give away for people who simply enter the contest in the US. But these are running out fast, so don’t waste any time with your entry.

The two programming-oriented contests in the challenge actually opened on the 1st of December in the US (there are separate contests in different countries). Entries can be sent in until the 31st of January. Good luck with your entries…

A few people have recently asked about the ability to ensure referential integrity in XML data. In this case, they are referring to the referential integrity feature that is common to relational databases. I thought I’d take a few moments to share one possible answer with you.

Referential integrity is a relational database feature that ensures consistency is maintained between items that reference one another. For instance, if you have a database table with order information that refers to a database table with product information, referential integrity ensures that each order refers to a valid product. In technical terms, referential integrity ensures that each product ID specified in the order table (where the product ID is a foreign key in the order table), exists in the product table.

Over the years, DBAs have found referential integrity to be a valuable feature in their relational databases. And now, they are asking how to ensure referential integrity with XML data.

The XML standard does not include mechanisms for ensuring referential integrity amongst XML elements. You can use schemas to place constraints on the XML data, but these constraints are not really enforced by the database (except insofar as the database is used to validate via a schema). And the ability to place constraints on XML data doesn’t quite add up to the ability to the ability to ensure the same level of consistency checking offered by referential integrity in a relational database.

However, there is a possible answer. This is one of the cases where using a hybrid relational/XML database proves to be very useful. You extract values from certain XML elements and store those values in relational columns in the same table, and then place constraints on those relational columns to ensure referential integrity. There is a cost associated with this approach, due to increased storage and additional programming logic. However, for many, this increased cost is justified by the ability to ensure referential integrity.

I thought I’d take a few moments and provide a quick report on the Information on Demand conference in Las Vegas. Like the past couple of years, it was a very interesting conference from a native XML storage point-of-view. There were all sorts of interesting sessions, including sessions about:

• Innovative uses of hybrid relation/XML modeling
• Native XML storage in SOA environments
• Native XML storage for electronic medical records
• Native XML storage for government information sharing
• Native XML storage for electronic forms
• And much, much more

For me, one of the highlights of the show was a session delivered by Li Cui and Charlie Wang from UCLA Health System. They described the different implementations of electronic medical records at UCLA Health System, and the benefits they are realizing from the adoption of native XML storage. Their presentation materials were simply outstanding, offering a very clear before and after indication of the impact of native XML storage. In their words, DB2 pureXML “significantly simplifies database design, web services development and robustly provides capability to handle XML data”. We heard about how it now takes hours instead of weeks to set up new types of electronic medical records, as well as a number of other quantifiable benefits. But it was the impact on the patients that left the greatest impression on me. It was great to hear how–thanks to their new system–patient records are now available much sooner, which obviously has a very real and significant impact on patient care.

Across all of the native XML storage sessions, some themes did emerge. One theme is that many organizations are using native XML storage to optimize their Service Oriented Architecture (SOA) environments. XML is the de facto language for information exchange, and there are a variety of use cases where organizations are persisting that XML in native format. These include the storage of transactional data, the implementation of a service bus cache, the logging of user events, and more. Another theme is that many organizations are using XML to augment traditional relational types. There are situations where modeling data using traditional relational types is challenging. One of those situations is where the structure of data changes often. In such cases, storing the fixed data in traditional relational columns and storing the data whose format changes often in XML format is an ideal solution. It takes advantage of the relative strengths of traditional relational types, as well as taking advantage of the flexible nature of XML for data whose format changes often.

I’m looking forward to the next conference… IOD Europe in Berlin next June.

As you saw in my previous post, IBM DB2 has spectacular performance numbers when benchmarking with XML data. As a reminder, DB2 performed 48.5 million transactions from 200 concurrent users over a two hour period on a server 4 CPUs, where each CPU had 6 cores. Admittedly, this is a relatively powerful server. But then again, you would expect to need a powerful server to process 48.5 million transactions in two hours. Today’s post will explain one of the key reasons why DB2 offers such impressive performance when compared to other major database vendors.

Most database vendors perform a similar set of steps when working with XML data. They parse the XML data, translate the XML data into a stream of tokens, logically represent the stream of tokens in a tree-like structure, store the tree-like structure, create indexes, and then manipulate and query the data as necessary. Of course, different vendors implement these steps in different ways, leading to some vendors enjoying a performance advantage in certain regards. For instance, some vendors have a higher reliance on large indexes than others, slowing data inserts, updates, and deletes. These vendors also typically require greater amounts of storage because of the need for these large indexes. However, it is query performance that concerns most people, so let’s focus on that.

Working with XML data poses an additional set of challenges when compared to working with traditional relational data. Many operations with XML data involve navigating around the tree-like structure that represents the XML data. For example, a query may impose a predicate on the values of the City and Age XML tags that appear inside a Customer tag, while returning the value of the Name tag that appears inside the same Customer tag. When you break this query into the atomic operations that are performed on the physical data, the query will involve many navigations of the tree-like structure. The database must navigate to the Customer element and then to the City element within it, apply the predicate, navigate to the Age element for that customer, apply the predicate, and if the predicates are satisfied, navigate to the Name element for that customer. And the database must perform these operations for many, many rows. Appreciating the need to optimize navigations of the tree-like representation of the XML data is one of the keys to understanding why IBM DB2 performs so well.

DB2 implements the tree-like structure in a way that allows you to very quickly navigate around the tree. Because DB2 stores pointers to both parent and child nodes for each element, and because DB2 optimizes the speed with which those navigations are performed, DB2 quickly navigates around the tree-like structures that represent the XML data. So, it is a combination of the physical representation of the tree-like structure, together with special algorithms for navigating around trees, that helps IBM achieve such blistering speed. After all, if you can optimize the atomic operations that underpin this breed of database queries, it only follows that you will enjoy the benefits at query time.

I do not have direct knowledge of the internal workings of other major relational database vendors. My opinions regarding how IBM manages XML data when compared with other major relational database vendors are drawn from information that is freely available on the internet. The opinions expressed in this post are mine and mine alone. They do not necessarily represent the opinions of my employer, IBM.

IBM continues to openly benchmark its DB2 pureXML capabilities.  Last week at the Information On Demand Conference in Las Vegas, Intel and IBM released details of their latest joint benchmark.  The goal of the benchmark is to show the performance levels that you can expect for inserts, updates, deletes, and queries on transactional XML data.  This latest benchmark is with 1TB of XML data using the FIXML standard industry format.  The results are very interesting.  Thanks to the way that DB2 stores XML data, the XML data occupies less than 450GB of disk space when stored in DB2.  If you compare the results of this benchmark against previous benchmarks, you can see that the addition of 50% more processing cores provides 48% more throughput (at lower CPU utilization rates), indicating a nice scale-out story.  Intel actually measured a 4-CPU server, with 6 cores per CPU, processing more than 6700 XML-based TPoX transactions per second.  One of the key aspects of the benchmark is the minimal amount of database tuning needed to obtain these results. For more details, including hardware specifications and database configuration, see the presentation from the conference.

Recently, I have had some conversations about the impact of native XML storage in DB2 for administrators of relational databases. You may be interested in the gist of those conversations.

Some DBAs are wary of changes to their relational environments. They have a finely-tuned environment that meets their service level agreements. I understand and appreciate their apprehension regarding “new features”. After all, its good to exercise caution when vitally important parts of the business depend upon your systems. However, you should be aware that, sometimes, you can reduce server load, improve query performance, or increase throughput by using native XML storage.

I cannot unequivocally tell you that using native XML storage will improve application performance in your environment, because your performance depends on many factors, both within the realm of the database and beyond. The one thing that I can unequivocally state is that a DBA who uses native XML storage can configure the storage of native XML data (tablespace, page sizes, buffers, etc) separately from the storage settings for other relational data and often improve performance.

For a good technical article on various aspects of performance for different approaches to storing XML data in DB2, see A performance comparison of DB2 9 pureXML and CLOB or shredded XML storage.

Some DBAs have expressed a fear that native XML storage may jeopardize their jobs. This fear stems from the perception that native XML storage eliminates tasks that currently occupy large amounts of time. For instance, there may no longer be a need to shred XML schemas into relational tables, to manage the many tables that may result from shredding a complex XML format, to update database table definitions after XML schema changes, and so on. It is true that these tedious tasks may be eliminated. However, many of the administrative tasks that are needed for relational data are still needed for native XML data. For instance, administrators must:

• Determine optimal page size for XML data
• Determine whether to set up separate table spaces
• Assess and set up buffer pools for XML
• Determine and implement an XML indexing strategy
• And so on

As you can see, the move to storing XML data in XML columns doesn’t eliminate the need for database administration. The need for fine-tuning the database remains as important as ever. The move to native XML storage does free DBAs from some of the tedium associated with setting up and updating shredding environments. Time that can possibly be spent on other tasks, or on tuning the system.

XML is here to stay. The amount of XML data that DBAs will need to manage will undoubtedly increase. After all, XML standards are increasingly emerging and being adopted. As such, it is in the best interests of DBAs to know and use the best way to manage that XML data. Native XML storage will not always be the answer to every XML storage question. Sometimes traditional relational storage methods will be best. However, one thing is certain: being able to determine the best option for XML data and being able to tune an environment with XML data for optimum performance is an increasingly valuable skill for DBAs.

My advice is… don’t be wary of native XML storage. Learn about it and know when to use it for your professional advantage.

Andrew Eisenberg is co-chair of the W3C XML Query Working Group. Andrew and I are delivering a free Webcast about XQuery. We will discuss XQuery, provide some tips for working with XQuery, and also talk about the future direction of XQuery. Make sure to sign up for this Webcast today.

Here are 10 reasons why relational database administrators need to understand and possibly use native XML storage…

I recently blogged about the Search for an XML Superstar contest. I’m happy to say that the Web site is now live at www.xmlchallenge.com.

The XML challange actually consists of five separate contests, so there is a good chance you will find a contest you can enter, regardless of your level of technical ability:

• Video contest
• Gadget contest
• Query contest
• Ported application contest
• XML contest

The contest is open to both students and professionals. You can enter as many of the contests as you like. In the US, prizes include Laptops, Wii consoles, Zune players, iPod Touches, iPod Nanos, USB keys, and T-shirts. Good luck with your entries…

Here are some observations about the XML storage capabilities in Oracle 11g. These observations were deduced from public sources. Please be aware that I work for IBM who competes directly with Oracle in this regard.

Oracle provides three options for storing XML data:

• Unstructured, which is essentially Character Large OBject (CLOB) storage. Like any CLOB implementation, you will need to retrieve and parse the XML data before executing XPath and XQuery statements, which has a query-time performance impact.
• XML-Object-Relational, which shreds the XML data into object-relational tables. There are multiple storage options to choose from (which I will cover in a moment). Oracle recommends this option for data-centric use cases. Of course, with this option, retrieving the original XML data will incur a performance hit as the data is re-composed.
• XML-Binary, which stores the XML data as a token stream in a Binary Large OBject (BLOB). Oracle recommends this option for document-centric use cases.

If you use XML-Object-Relational, which is also known as Structured or Schema-based storage, you have five different options to choose from for storing repeating elements:

• Store in CLOBs
• Store in VARRAY as LOBs
• Store in VARRAY as nested tables
• Store in VARRAY as XMLType pointers to BLOBs
• Store in VARRAY as XMLType pointers to nested tables

If your head is spinning with the different options, I don’t blame you.

Oracle provides a special index called XMLindex, which indexes the internal structure of XML data. Actually, this index is a table. There is an interesting post on the Oracle Discussion Forum. In this post, an Oracle user describes their experiments with XBRL data in Oracle 10g and 11g. In their experiments, case 3 runs on Oracle 11g without XMLIndex in 9.6 seconds, while case 4 runs with XMLIndex and takes 574 seconds. So, in this user’s experience, running with the XMLIndex results in approximately a 50x slowdown.

Note that if you want to create an index for numeric or date values in Oracle 11g, you must use stored procedures, which create separate indexes that you must asynchronously maintain.

Oracle now has an XML update function. It is not compliant with the W3C XQuery Update Facility. Although, Oracle did recently announce the XQuilla XQuery engine, which they claim “will (maybe) free the way for the W3C XQuery Update Facility 1.0 candidate specification / implementation, which is embedded in the XQuilla XQuery engine, for other Oracle products”. So it is possible that a future release of Oracle may support the W3C XQuery Update Facility.

The compendium of XML storage options in Oracle 11g are, in my opinion, essentially based upon existing relational infrastructure. This is very different than the approach that IBM has taken, where they have truly built native support for XML data into their database from the ground up.

Please note that these are solely my personal opinions and not necessarily those of my employer IBM.