XML

Processing XML with PowerShell II

Dan Sullivan — Tue, 28 Nov 2006 13:12:00 GMT

In our previous blog article Processing XML with PowerShell we looked at using XPath expressions to do calculations that used XML as input. One of the things that this article pointed out was that you often can do an entire calculation within an XPath expression. Sometimes, however, you want to read the XML file, pull parts out of it and process them outside of the XML file. One of the ways of doing this is to use the dotted syntax and the Item ParameterizedProperty features of PowerShell so that you can treat XML as an object.

We are going to start off by looking at using the PowerShell object model for XML to pull apart an XML file so that we can process it, then we are going to look at an extension function aliased as xitems to do the same thing, but typically with less effort and more capabilities. Lastly we will look at how the xitems function is implemented. You can download the script for making this function and the sample files at http://www.pluralsight.com/dan/samples/ProcessingXMLPowershell-2.zip. Note that this will include the XSLT.ps1 script, updated for the xitems function, that was include in the ProcessingXMLPowerShell.zip file.

We will start by looking at a file named Stock.xml. The Stock.xml file looks like:

<GroceryList
ID = "A-24"
>
<Stock>
    <Dept>
    <Area>Round</Area>
    <Name>Produce</Name>
    </Dept>
    <Name>Orange</Name>
<Price>3.41</Price>
</Stock>
<Stock>
    <Dept>
    <Area>Beef</Area>
    <Name>Meat</Name>
    </Dept>
    <Name>Steak</Name>
    <Price >13.20</Price>
</Stock>
<Stock>
    <Dept>
    <Area>Leaf</Area>
    <Name>Produce</Name>
    </Dept>
    <Name>Lettuce</Name>
    <Price>1.36</Price>
</Stock>
</GroceryList>

Note that each Dept element includes both an Area and Name element. We want to find out the names of the departments in the stock.xml file. We can do this by piping all of the department names into a select-object -unique cmdlet. Here is a script that does just that.

PS C:\Demos> [xml]$s = get-content C:\Demos\Stock.xml
PS C:\Demos> $s.GroceryList.Stock | %{$_.Dept.Name} | select-object -unique
Produce
Meat
PS C:\Demos>

Here we are using the dotted syntax to extract each Stock element from the stock.xml file and pipe it into a following pipeline segment. In that second pipeline segment we use the dotted syntax again to extract the Name from the Dept element of current pipeline object; The current pipeline object in this case is the Stock element. Then we pipe all the names we have found into the select-object cmdlet that makes a unique list of those names.

It’s easy to come up with a rather wordy description of what this script is doing; It is saying something like “Give me all of the elements named GroceryList at the root of the $s XML document; then for each one of these get me all of its children whose name is Stock; then for each one of these get me all of its children whose name is Dept; then for each on of these get me all of its children whose name is Name.” This sort of description could be applied to almost anything that manages hierarchical data including XPath, which we will be looking at later when we examine the xitems function.

Thinking of this hierarchical description makes it seem that the script below would be alternate, more simple, way to find the names of the departments.

PS C:\Demos> [xml]$s = get-content C:\Demos\Stock.xml
PS C:\Demos> $s.GroceryList.Stock.Dept.Name | select-object -unique
PS C:\Demos>

This, seemingly obvious, way of getting the children of the children, etc. did not produce any results. The dotted syntax is somewhat limited because of the way PowerShell models XML. If we take closer look at what is actually being returned for the GroceryList and Stock elements we will see why.

PS C:\Demos> $s.GroceryList.GetType()
IsPublic IsSerial Name       BaseType
-------- -------- ----      --------
True     False    XmlElement System.Xml.XmlLinkedNode
PS C:\Demos> $s.GroceryList.Stock.GetType()
IsPublic IsSerial Name      BaseType
-------- -------- ----      --------
True     True     Object[]  System.Array
PS C:\Demos>

The type of the GroceryList is an XmlElement as you might suspect, after all we are working with XML. However the type of Stock is Array, not an XmlElement or XmlElement[], and that is why it lacks a Dept property. Now let’s look at what happens when instead of trying to access the Dept element from Stock we pipe the array into another pipeline segment and see what the type is.

PS C:\Demos> $s3.GroceryList.Stock | %{$_.GetType()}
IsPublic IsSerial Name        BaseType
-------- -------- ----        --------
True     False    XmlElement  System.Xml.XmlLinkedNode
True     False    XmlElement  System.Xml.XmlLinkedNode
True     False    XmlElement  System.Xml.XmlLinkedNode
PS C:\Demos>

It turns out that the Stock array is an array of XmlElements and piping it into a pipeline segment enumerates that array. The result is that inside the pipeline segment $_ is an XmlElement that has a Dept child element.

So to drill into an XML hierarchy using the PowerShell object model you must break up what seems like a natural dotted syntax into pipeline segments, one pipeline segment for every two levels of depth you want to go into the XML hierarchy. The word description earlier of what is happening here, however, is in effect the definition of an XPath construct called a LocationPath.

We used XPath expressions in Processing XML with PowerShell to do processing of an XML file. XPath is really a simple language, an expression can only produce one of four datatypes; A number, a string, a boolean, or a node set. We were using those first three scalar types to do calculations with the xeval function.

A node set is what the name seems to imply, it is a set of XML nodes. It might be a set of XML elements or attributes or a mixture of these an other kinds of XML nodes. The data model in the XPath Recommendation defines seven kinds of nodes that might be found in an XML documents. A LocationPath is an XPath expression that produces a node set instead of a scalar value. It is given a special name because it is such a common idiom to use XPath to produce a node set. Here is a LocationPath that will produce a nodeset that consists of all of the Name elements from the stock.xml file:

GroceryList/Stock/Dept/Name

We can use this LocationPath with the xitems function to find the names of the departments in the stock.xml file, as we did at the beginning of this article.

PS C:\Demos> xitems C:\Demos\Stock.xml "GroceryList/Stock/Dept/Name" |
 select-object -property value -unique
Value
-----
Produce
Meat
PS C:\Demos>

The xitems function requires at least two parameters. The first is the path to the XML file we want to process, or as we will later see an XPathNavigator. The second argument is an XPath LocationPath. You can see we are able to use a more simple model to specify the parts of the XML file we wish to process.

In the Processing XML with PowerShell blog article we looked at processing XML files that contained namespaces and some of the issues you can run into when using the PowerShell object model of XML. The xitems function uses the same technique as the xeval did, you pass a dictionary that contains the mapping of prefixes and namespaces to it. Here is another file that uses namespaces, stockNS.xml.

<GroceryList xmlns='urn:prices'
xmlns:loc='urn:location'
xmlns:ident='urn:identity'
ID = "A-24"
>
<Stock xmlns="urn:inventory">
    <loc:Dept>3rd floor</loc:Dept>
    <ident:Dept>
    <Area>Round</Area>
    <Name>Produce</Name>
    </ident:Dept>
    <Name>Orange</Name>
<Price>3.41</Price>
</Stock>
<Stock xmlns="urn:inventory">
    <loc:Dept>2nd floor</loc:Dept>
    <ident:Dept>
    <Area>Beef</Area>
    <Name>Meat</Name>
    </ident:Dept>
    <Name>Steak</Name>
    <Price >13.20</Price>
</Stock>
<Stock xmlns="urn:inventory">
    <loc:Dept>3rd floor</loc:Dept>
    <ident:Dept>
    <Area>Leaf</Area>
    <Name>Produce</Name>
    </ident:Dept>
    <Name>Lettuce</Name>
    <Price>1.36</Price>
</Stock>
</GroceryList>

This file uses quite a few namespaces, just to make things interesting and because typically when namespaces are used they are often used a lot. Lets do our department names calculation again, using the PowerShell xml object model.

PS C:\Demos> [xml]$s = get-content C:\Demos\StockNS.xml
PS C:\Demos> $s.grocerylist.stock 
  | %{$_.Item("Dept", "urn:identity").Name} | select-object -unique
Produce
Meat
PS C:\Demos>

Here we have used the Item property that PowerShell adds to an XML element to make it possible to access elements that are distinguished by their namespace. Here is a script that uses xitems to get the same results:

PS C:\Demos> xitems C:\Demos\StockNS.xml `
  "p:GroceryList/i:Stock/id:Dept/i:Name" `
  @{p="urn:prices";i="urn:inventory";id="urn:identity"} |
  select-object -property value -unique
Value
-----
Produce
Meat

PS C:\Demos>

You might look at this and say that the PowerShell object model for XML is in at least one respect much easier to use than one based on XPath that xitems uses because in the it does not require the specification of the namespaces for the GroceryList, Stock and Name elements. However namespaces are part of an XML file for a reason, to make sure that names in common usage can easily be distinguished. Look at this alternate version of the StockNS.xml file:

<GroceryList xmlns='urn:prices'
xmlns:loc='urn:location'
xmlns:ident='urn:identity'
ID = "A-24"
>
<Stock xmlns="urn:inventory">
    <loc:Dept>3rd floor</loc:Dept>
    <ident:Dept>
    <Area>Round</Area>
    <Name>Produce</Name>
    </ident:Dept>
    <Name>Orange</Name>
<Price>3.41</Price>
</Stock>
<Stock xmlns="urn:ignore">
    <loc:Dept>2nd floor</loc:Dept>
    <ident:Dept>
    <Area>Beef</Area>
    <Name>Meat</Name>
    </ident:Dept>
    <Name>Steak</Name>
    <Price >13.20</Price>
</Stock>
<Stock xmlns="urn:inventory">
    <loc:Dept>3rd floor</loc:Dept>
    <ident:Dept>
    <Area>Leaf</Area>
    <Name>Produce</Name>
    </ident:Dept>
    <Name>Lettuce</Name>
    <Price>1.36</Price>
</Stock>
</GroceryList>

Notice that the second Stock element is in the “urn:ignore” namespace, not the “urn:inventory. The PowerShell dotted syntax would have included this element in its selection of department names, which may not be what was really intended. In order to be sure what you are processing XML in the namespace you intend with the PowerShell XML object model you really have to use the Item method at every level of the XML hierarchy. Your mileage may vary, but using XPath to select items from an XML file will in general be easier than using the PowerShell object model of XML and more capable.

An XPath LocationPath can be thought of as a filter; You use it to filter out the parts of the document you are not interested in. This filter can be about as fine-grained as you would like. For example what if we wanted the department names on the 3rd floor?

PS C:\Demos> xitems C:\Demos\StockNS.xml `
   "p:GroceryList/i:Stock[loc:Dept='3rd floor']/id:Dept/i:Name" `
   @{p="urn:prices";i="urn:inventory";id="urn:identity";loc="urn:location"} |
   select-object -property value -unique
Value
-----
Produce
PS C:\Demos>

The LocationPath used by this script has a predicate in it, that only selects Stock elements that have a loc:Dept child element whose value is “3rd floor”.

The xitems function is produces an XPath navigator, and it can also take a XPathNavigator as input. This means you can use the results of one xitems function as input to another. Here is an example using stock.xml, the file without the namespaces to reduce the clutter:

PS C:\Demos> xitems C:\Demos\Stock.xml "GroceryList/Stock" |
  %{xitems $_ "Dept/Name"} | Select-Object -property value -unique
Value
-----
Produce
Meat

PS C:\Demos>

This particular script is analogous to the first script that we presented in this article. We’ve broken the selection into two pipeline segments just to show that the second segment could use the output of the first as input. The first pipeline segment pipes an XPathNavigator into the second pipeline segment which uses that XPathNavigator as input to another xitems function.

From the Processing XML with PowerShell we know that xeval can also process an XPathNavigator, so the output of the xitems function can also be passed into the xeval function. Let’s try that:

PS C:\Demos> xitems C:\Demos\Stock.xml "GroceryList/Stock" |
  %{xeval $_ "string(Dept/Name)"} | Select-Object  -unique
Produce
Meat
PS C:\Demos>

In this example the second pipeline segment evaluates the result of the first pipeline segment. Note that in the third segment the Select-Object cmdlet is not using the -property value option. That is because the second segment is producing a string and a string does not have a value property.

Lastly xitems is similar to xeval in that you can pass it an array of LocationPaths and it will apply all of them to an XML file.

PS C:\Demos> xitems C:\Demos\Stock.xml "GroceryList/Stock/Dept/Area",
  "GroceryList/Stock/Dept/Name" | Group-object -property value
Count Name                      Group
----- ----                      -----
    1 Round                     {Area}
    2 Produce                   {Name, Name}
    1 Beef                      {Area}
    1 Meat                      {Name}
    1 Leaf                      {Area}
PS C:\Demos>

This may seem a strange query, but it does show us, for example, that there are two Stock elements that have Name children whose value is “Produce”.

So far we have seen the basics of using the xitems function and that it shares much in common with xeval. Let’s now take a look at the implementation of xitems.

First of all xitems is an alias for get-XSLT_XPathSelection. As the names implies this function is making an XPath selection.

filter get-XSLT_XPathSelection
{
param($nav, [array]$expressions, [hashtable]$namespaces)
if($nav -is [string])
{
$nav = get-XSLT_XPathNavigator $nav
}
if($nav -isnot [System.Xml.XPath.XPathNavigator]) 
{ throw "String path or XPathNavigator required"}
$nm = get-XSLT_NamespaceManager $nav.NameTable $namespaces
$xpathExpression = "";
foreach($exp in $expressions)
{
if($xpathExpression -ne "")
{
$xpathExpression += " | ";
}
$xpathExpression += $exp
}
$nodes = $nav.Clone().Select($xpathExpression, $nm);
$nodes;
}

The xitems function starts off the same as the xeval function that we looked at in the Processing XML with PowerShell article, it has a parameters an untyped $nav, an array and a hashtable. Just as xeval does, xitems converts a string to an XPathNavigator. It then builds a namespace manager and iterates through the expressions that were passed in, just like xeval does. In fact the only real differences from xeval is that xitems concatenates the selection expressions using the XPath alternate operator, “|” and uses Select instead of Evaluate on the XPathNavigator that was passed in.

So in conclusion we can see that using xitems is really easier and more consistent than using the dotted syntax and Item method that the PowerShell XML object model uses, and is a lot more capable. Of course you will have to learn about XPath to fully exploit those capabilities, but it will be well worth you effort doing so.

Processing XML with PowerShell

Dan Sullivan — Sat, 25 Nov 2006 11:25:00 GMT

There are a couple of powerful technologies for processing native XML, XPath and XSLT. People often avoid processing native XML but instead convert the XML to an object model in a language they are used to and do “conventional” programming on that model. Even PowerShell itself does this with its fairly straightforward dotted syntax for accessing parts of an XML document and of course .NET, web service technologies, and SQL Server have their own ways to morph XML into a familiar object model.

There are probably a number reasons for this not the least of which is syntactic comfort… with some practice you actually can drive nails with a screwdriver and then you only need to learn how to use one tool to build a house. XSLT itself is often criticized as being too verbose but that is not really the case. And lastly the programming models for XPath and XSLT are different than that used in languages like C# or VB.NET; They are much more like SQL in that you don’t actually write a program but instead write a set of rules and throw data at them.

However if you are going to bump into XML in your travels, and you can be pretty sure that you will, it is really worth your while to become comfortable with at least the basics of XPath and XSLT because that knowledge will make a lot of programming jobs a lot easier. Let’s take a look at a simple example to see this. Here is a grocery list, XML-style, in the file groceries.xml

<GroceryList>
<Item>
<Dept>Produce</Dept><Name>Orange</Name><Price>3.20</Price>
</Item>
<Item>
<Dept>Meat</Dept><Name>Steak</Name><Price>13.20</Price>
</Item>
<Item>
<Dept>Produce</Dept><Name>Lettuce</Name><Price>1.34</Price>
</Item>
<Item>
<Dept>Meat</Dept><Name>Ham</Name><Price>11.41</Price>
</Item>
</GroceryList>

We can calculate the total of all of groceries using the PowerShell object model of XML with the following script;

PS C:\Demos> [xml]$list = get-content .\groceries.xml
PS C:\Demos> $list.GroceryList.Item | &{begin {$sum=0}
 process{$sum += $_.Price} end {$sum}}  
29.15
PS C:\Demos>

There are other ways to do this in PowerShell, but all involve iterating through the items to produce a sum. It turns out there is a simple XPath expression that calculates sum of the prices of the items in the list:

sum(GroceryList/Item/Price)

In fact it would be kind of nice if we had a way to “execute” and XPath expression easily in PowerShell. How about this?

PS C:\Demos> xeval groceries.xml "sum(GroceryList/Item/Price)"
29.15
PS C:\Demos>

This blog article is about processing XML using PowerShell and the typical sorts of things you run into when you do this. It uses some extension functions, xeval and xnav are their aliases, to do this processing. The xeval function is used to process an XML file using XPath expressions. The xnav function is used to turn literal XML into an XPathNavigator. Later blog articles will cover other ways to process XML using PowerShell.

A script to build these functions and their associated aliases is in a file named XSLT.ps1. This file and the examples in this blog article are available at http://www.pluralsight.com/dan/samples/ProcessingXMLPowershell.zip. These extension functions are not really any harder to use than the XML support built into PowerShell but are quite a bit more capable in what they can accomplish. After we look at using these extension functions we will look inside of XSLT.ps1 and see how it works.

The XSLT.ps1 file actually has some other extension functions that are not discussed in this blog article but will be in a future one.

In the first example of using xeval we just looked at, the first argument to the xeval function is the file path for the XML file you want to process. The second argument is the XPath expression you want evaluated. Of course to make good use of xeval you will have to be familiar with XPath. XPath is a W3 recommendation and is at http://www.w3.org/TR/xpath.

The XPath recommendation is certainly worth reading and contains many example of XPath expressions. Another good source to have at your side is “Essential XML Quick Reference” published by Addison-Wesley and written by Aaron Skonnard and Martin Gudgen.

Let’s start by looking at one of the issues you run into when working with XML. XML is often treated as though it were text and that is how PowerShell treats it. But XML is not plain ol’ text and the following examples will show that. We have another version of our xml grocery list in a file named GroceriesUC.xml. Let’s use our PowerShell script to process it.

PS C:\Demos> [xml]$list = get-content groceriesuc.xml
Cannot convert value "System.Object[]" to type 
"System.Xml.XmlDocument". 
Error: "Root element is missing."
At line:1 char:11
+ [xml]$list  <<<< = get-content GroceriesUC.xml
PS C:\Demos>

Hmmmm, that generated an error. What’s going on here?

We often think of files as containing text, that is the characters we see on the printed page. But files don’t contain text, the are just a sequence of bytes. When someone gives you a “text” file you must know how that text was encoded into a sequence of bytes in order to be able to read it. PowerShell gives us a little help here in that the get-content cmdlet lets you specify the encoding of the file if you know it, or “unknown” if you don’t. Well we don’t know the encoding of the file so let’s tell PowerShell that the encoding is unknown and see what happens.

PS C:\Demos> [xml]$list = get-content GroceriesUC.xml -encoding unknown
Cannot convert value "????????????????????????????????????????????
?????????????????????????????????????????????????????????????????????
 to type "System.Xml.XmlDocument". Error: 
 "Data at the root level is invalid. Line 1, position 1."
At line:1 char:11
+ [xml]$list  <<<< = get-content GroceriesUC.xml -encoding unknown
PS C:\Demos>

Looks like we are out of luck here too. It turns out the encoding of the file is UTF-16BE. That’s a standard encoding used for XML files that is a sequence of words with the high-order byte of the word coming first. You might see it in XML that is generated on non-Intel compatible processors. Now that we know that we know actual encoding we can pass the information onto PowerShell.

PS C:\Demos> [xml]$list = get-content GroceriesUC.xml 
    -encoding BigEndianUnicode
PS C:\Demos> $list.GroceryList.Item | 
    &{begin {$sum=0} process{$sum += $_.Price} end {$sum}}
29.15
PS C:\Demos>

Bottom line is when it comes to text unless you know the actual encoding you can’t depend on being able to read it. Earlier we said that XML wasn’t really text. To see what this means lets try that xeval function again on the GroceriesUC.xml file.

PS C:\Demos> xeval GroceriesUC.xml "sum(GroceryList/Item/Price)"
29.15
PS C:\Demos>

It works just fine and we don’t have to tell it what the encoding is. The reason for this is a requirement of every XML processor, i.e. a support library for XML such as the one in .NET that xeval uses, must be able to unambiguously figure out the encoding used in an XML file without any outside help. This is thought by many to be the key feature of XML and certainly is one of the reasons for its wide use today. It works so well that most people don’t even know it is a feature!

The built in processing in PowerShell using get-content makes a non-compliant XML processor. In some cases this isn’t that important but you should keep in mind that in the general case it is not useful for processing XML. If you want do know the details of how this “self-encoding” in XML works there is an explanation of it in Appendix F of the W3 Extensible Markup Language XML recommendation at http://www.w3.org/TR/xml/.

There is another issue that comes up when you deal with XML, namespaces. There are some who feel that namespaces are an unnecessary complication to XML, but they are important enough to have their own specification, Namespaces in XML which is at http://www.w3.org/TR/xml-names/. For those with an interest in such things the Extensible Markup Language XML is really just a grammar with a little over 80 productions with lots of comments in it, and Namespaces in XML just adds a few productions to that grammar. Regardless of how you feel about namespaces you will have to deal with them. Here is a different version of our grocery list. It is in the file named GroceriesNS.xml.

<GroceryList xmlns="urn:foo"
xmlns:r="urn:retail"
xmlns:w="urn:wholesale" >
<Stock>
<Dept>Produce</Dept><Name>Orange</Name>
<w:Price>3.20</w:Price><r:Price>4.20</w:Price>
</Stock>
<Stock>
<Dept>Meat</Dept><Name>Steak</Name>
<r:Price >14.20</r:Price><w:Price >13.20</w:Price>
</Stock>
<Stock>
<Dept>Produce</Dept><Name>Lettuce</Name>
<w:Price>1.34</w:Price><r:Price>2.34</r:Price>
</Stock>
<Stock>
<Dept>Meat</Dept><Name>Ham</Name>
<w:Price>11.41</w:Price><r:Price>14.41</r:Price>
</Stock>
</GroceryList>

This file is different from the Groceries.xml file in three ways. One is that it uses namespaces. Another is that it contains both a wholesale and a retail price for each item. It also uses Stock elements instead of Item elements; We will see why in a second. The Price elements are distinguished by their namespace, the ones prefixed with “r” are retail prices. The prices in the r:Price elements are the same as the corresponding Price elements in the Groceries.xml file. Let’s use PowerShell’s object model of XML to calculate the sum of the retail prices. PowerShell sees two Price elements under the Stock element, so it makes an array out of them. We will have pick which one to sum up.

PS C:\Demos> $list = get-content GroceriesNS.xml
PS C:\Demos> $list.GroceryList.Stock | 
    &{begin {$sum=0} process{$sum += $_.Price[1]} end {$sum}}
30.15
PS C:\Demos>

The retail prices in the GroceriesNS.xml file are the same as the ones in the unqualified prices in the Groceries.xml file so we should get the same answer as before, but we don’t. The problem we have run into is that Price elements are distinguished only by their namespace and not by there position in the file. Note that in the second Stock element in the file the wholesale price comes after the retail price. So we have to make sure that we pick the correct price element.

To distinguish a Price element we have to use a ParameterizedProperty named Item that PowerShell adds to an XML element. In many cases you will find it difficult to process XML using the PowerShell object model if the XML contains any Item elements because PowerShell uses this name for the ParameterizedProperty it adds to XML elements. This is why we changed the name of the Item element to Stock. If we had not made this change we would not have been able to process this XML file using the PowerShell object model of XML.

In any case the Item property allows us to specify both the name and the namespace of the element we want.

PS C:\Demos> $list = get-content GroceriesNS.xml
PS C:\Demos> $list.GroceryList.Stock | 
  %{$_.Item("Price", "urn:retail")} | 
  &{begin {$sum = 0} process {$sum += $_.get_InnerText()} end {$sum}}
29.15
PS C:\Demos>

Now we get the 29.15 just as we did when we processed the Groceries.xml file.

Now let’s do the same thing using xeval function.

PS C:\Demos> xeval GroceriesNS.xml 
"sum(a:GroceryList/a:Stock/r:Price)" @{r="urn:retail";a="urn:foo"}
29.15
PS C:\Demos>

In this example xeval function has a third argument that is a dictionary that maps prefixes to the namespaces they represent in the XPath expression. You can see the GroceryList and Stock end up in the “urn:foo” namespace because of the “a” prefix and likewise Price ends up in the “urn:retail” namespace. Note that the prefix used in the XPath expression is not necessarily the same as that in the source XML file. There is no requirement the prefix used in an XPath expression be the same as that in the source XML file being processed; The key thing is that is specifies the proper namespace. Note that in the GroceriesNS.xml file the GroceryList and Stock element had no prefix but that the default namespace for the file was “urn:foo”.

Let’s look at some more things we can do with xeval. The second parameter of xeval may be an array of XPath expressions. xeval will evaluate each of these expressions.

PS C:\Demos> xeval GroceriesNS.xml "sum(a:GroceryList/a:Stock/r:Price)",
    "count(a:GroceryList/a:Stock)" @{a="urn:foo";r="urn:retail"}
29.15
4
PS C:\Demos>

Here we calculated the sum of the retail prices and number of Stock items. Note that this example makes use of the fact that in PowerShell the “,” operator makes an array of the arguments it joins. Let’s carry this one step further.

PS C:\Demos> xeval GroceriesNS.xml "sum(a:GroceryList/a:Stock/r:Price)",
    "count(a:GroceryList/a:Stock)",
    "sum(a:GroceryList/a:Stock/r:Price)
    div count(a:GroceryList/a:Stock)" @{a="urn:foo";r="urn:retail"}
29.15
4
7.2875
PS C:\Demos>

Here, beside the sum of the prices and the number of stock items, we calculate the average price of the stock items. The important point of these last few examples is that it is very common to calculate some value based on the content of an XML file. These calculations can be embedded in an XPath expression and you never have to “read”, i.e. pull out and interpret parts of, the XML file to do this.

You might think that all repeated a:GroceryList etc. might be inefficient or at least is tedious. First of all it’s not really inefficient at all to calculate a path multiple times in an XPath expression because the XPath engine that is evaluating these expression caches paths and reuses them when they appear again. As far as the tedium of typing them multiple times you can leverage PowerShell itself to simplify that.

PS C:\Demos> $s = "a:GroceryList/a:Stock"
PS C:\Demos> $p = "$s/r:Price"
PS C:\Demos> xeval GroceriesNS.xml "sum($p)",
    "count($s)",
    "sum($p) div count($s)" @{a="urn:foo";r="urn:retail"}
29.15
4
7.2875
PS C:\Demos>

Here we have made use of the fact that PowerShell will build a string out of a combination of literal text and variables. If the format of the XML file is pretty regular you can make the XPath expression used for the evaluation even more simple.

PS C:\Demos> xeval Groceries.xml "sum(//r:Price)" @{r="urn:retail"}
29.15
PS C:\Demos>

Of course here again you need some knowledge of XPath to simplify things. The “//” part of the XPath expression in this case really means “Find all the r:Price elements in the file.”

There is a hidden value in using XPath expressions to do calculations on an XML file; That expression can be used by anyone using any technology that implements XML support to do the same calculation on that file. In other words the XPath expression is a platform independent way of specifying how a calculation is done, it is not limited to PowerShell.

Sometimes you will have a literal string for your xml instead of a file. You can’t pass this directly to the xeval function because it will interpret that string as a file path and attempt to load a file.

The implementation of xeval internally uses an XPathNavigator to process the XML that is passed to it. This blog article isn’t going discuss the details of how XPathNavigator works, but xnav is an alias for a function that converts literal XML into an XPathNavigator. If the first parameter passed into xeval is an XPathNavigator it will use that navigator instead of interpreting it as a file path.

Here is an example of processing literal XML.

PS C:\Demos> $nav = xnav "<Stock><sku>ee-44</sku></Stock>"
PS C:\Demos> xeval $nav "string(//sku)"
ee-44
PS C:\Demos>

This example begins by using the xnav function to make an XPathNavigator out of some literal XML. This XPathNavigator is passed into the eval function. The XPath expression passed to xeval pulls out the stockroom unit from the literal XML.

Using the pipeline in PowerShell is a great way process XML. There are a number of grocery files with names like GroceriesNS1.xml, GroceriesNS2.xml and so on that we would like to process. We would like to calculate the value of each these files. This is what the GroceriesNS1.xml file looks like.

<GroceryList xmlns="urn:foo"
xmlns:r="urn:retail"
xmlns:w="urn:wholesale" 
ID = "A-24"
>
<Stock>
<Dept>Produce</Dept><Name>Orange</Name>
<w:Price>114.20</w:Price><r:Price>3.41</r:Price>
</Stock>
<Stock>
<Dept>Meat</Dept><Name>Steak</Name>
<r:Price >13.20</r:Price><w:Price >14.20</w:Price>
</Stock>
<Stock>
<Dept>Produce</Dept><Name>Lettuce</Name>
<w:Price>21.34</w:Price><r:Price>1.36</r:Price>
</Stock>
</GroceryList>

First of all it’s straightforward to get the names of these files.

PS C:\Demos> get-childitem C:\Demos\* |
 ?{$_.Name -match "GroceriesNS\d+.xml"}
    Directory: Microsoft.PowerShell.Core\FileSystem::C:\Demos

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---        11/25/2006  10:15 AM        438 GroceriesNS1.xml
-a---        11/25/2006  10:14 AM        324 GroceriesNS2.xml
-a---        11/25/2006  10:16 AM        438 GroceriesNS3.xml

PS C:\Demos>

Note that the GroceryList element has an attribute name ID that identifies that list. We want to include that ID in our results.

PS C:\Demos> get-childitem C:\Demos\* |
 ?{$_.Name -match "GroceriesNS\d+.xml"} |
 %{xeval "$_" "string(f:GroceryList/@ID)", "sum(//r:Price)" `
 @{f="urn:foo";r="urn:retail"} }
A-24
17.97
31
54.4
109
57.97
PS C:\Demos>

In this example we pipe the file names into a script block that uses the xeval function. This uses XPath expressions to get both the ID of the GroceryList and the sum of its Price elements. Note the backtick and the end of the third line to insure the continuation of the command line.

The output we get is ID followed by sum. We might like something that produces a single line per GroceryList. We could pipe these results into another script block that aggregated these results by the pair… or we could use XPath to do the same thing.

PS C:\Demos> get-childitem C:\BlogArts\ProcessingXMLPowerShell\* |
 ?{$_.Name -match "GroceriesNS\d+.xml"} |
 %{xeval "$_" "concat(string(f:GroceryList/@ID), ' : ', sum(//r:Price))" `
 @{f="urn:foo";r="urn:retail"} }
A-24 : 17.97
31 : 54.400000000000006
109 : 57.97

Here we use the XPath concat function to produce a line per GroceryList report of the sum of the prices of each grocery list. You can produce some pretty fancy reports using just XPath expressions, but if they are much more complicated than the one in this example you will find it somewhat tedious to code them up. For more complicated reports XSLT is really a better choice and we will be looking at that in a later blog article. In any case this example has defined a report in terms of an XPath expression which anyone on any platform that implements XML can produce the same report. This example didn’t “code up” a report it made a rule that defined how the report was to be produced.

Now let’s look at the implementation. We will start with the eval function.

filter get-XSLT_XPathEvaluate
{
param($nav, [array]$computations, [hashtable]$namespaces)
if($nav -is [string])
{
$nav = get-XSLT_XPathNavigator $nav
}
if($nav -isnot [System.Xml.XPath.XPathNavigator]) 
  { throw "String file path or XPathNavigator required"}
$nm = get-XSLT_NamespaceManager $nav.NameTable $namespaces
foreach($n in $nav)
{
foreach($compute in $computations)
{
$n.Clone().Evaluate($compute, $nm)
}
}
}
set-alias xeval get-XSLT_XPathEvaluate

The xeval function uses three parameters. The first is a string or an XPathNavigator, the second is an array of XPath expressions, and the last is a dictionary of namespace mappings. It tests the first parameter to see if it is a string. If it is it uses the get-XSLTXPathNavigator function to make an XPathNavigator from the file path. We will look at the get-XSLTXPathNavigator function shortly.

Next it checks to make sure that the $nav variable is in fact an XPathNavigator and throws an error if it isn’t.

In order to use namespace with an XPathNavigator you need a construct called an XmlNamespaceManager. This construct holds the mappings of prefixes to namespaces. Both XPathNavigators and XmlDocuments store their associated XML in a non-textual, binary form for efficiency. Internally another construct, a NameTable, maintains a mapping between the names of elements and attributes, and their internal representation. The XmlNamespaceManager uses this NameTable in its constructor so that it can have the same mapping of names to internal representation that the XPathNavigator does.

Once the XmlNamespaceManager is constructed it is filled by get-XSLT_NamespaceManager function that we will look at shortly.

To do the computations the xeval function iterates through the array of XPath expression that are passed in. It uses a clone of the XPathNavigator to execute the expression. The reason it uses a clone of the XPathNavigator is the XPathNavigator is really a cursor on the XML file and we want to leave that XPathNavigator in its original state for each execution of XPath expressions being processed.

filter get-XSLT_XPathNavigator
{
param ($xml)
if($xml -is [string])
{
$xml = get-XSLT_XMLReader $xml;
$xml = get-XSLT_XPathDocument $xml
}
$nav = $xml.CreateNavigator();
$nav
}

The get-XSLT_XPathNavigator uses the string passed into it as a file path. It starts by converting the file path into an XmlReader, then uses that XmlReader to make an XPathDocument, which in turn is used to make an XPathNavigator.

filter get-XSLT_XMLReader
{
param ([string]$xmlFile)
[System.IO.FileStream]$fileStream = new-object System.IO.FileStream $xmlFile, 
    ([System.IO.FileMode]::Open),
([System.IO.FileAccess]::Read)
[System.Xml.XmlTextReader]$rdr = new-object System.Xml.XmlTextReader $fileStream
$rdr
}

The get-XSLT_XMLReader function opens a FileStream using the string passed in as the path to the file. Note that it is not using a StreamReader which would convert the file to text, it is instead reading the raw bytes in the file. The FileStream is used to make an XmlTextReader. Again, dispite its name, an XmlTextReader does not read text, it reads bytes from the FileStream and because it is a complient XML processor it is completely capable of determining the encoding of the XML that is in the byte stream.

filter get-XSLT_XPathDocument
{
param ([System.Xml.XmlReader]$xml)
$doc = new-object System.Xml.XPath.XPathDocument $xml;
$doc
}

The get-XSLT_XPathDocument function uses an XmlReader to make an XPathDocument. An XPathDocument is, in effect, a readonly XmlDocument except that the only thing you can do with it is make an XPathNavigator out of it. If all you are going to do is read the content of an XML file and not modify it, and XPathDocument may be a better choice because it may be more efficient at processing XPath than the XmlDocument.Select method is.

function get-XSLT_NamespaceManager 
([System.Xml.NameTable] $nameTable, [hashtable] $namespaces)
{
$nm = new-object System.Xml.XmlNamespaceManager $NameTable
foreach($key in $namespaces.keys)
{
$nm.AddNamespace($key, $namespaces.$key);
}
,$nm
}

The get-XSLT_NamespaceManager has two inputs, a NameTable and a dictionary of namespace mapping. It starts by making an XmlNamespaceManager. It then iterates through the keys in the dictionary and uses the key and it associated value to add namespace mappings to the XmlNamespaceManager. Note that it uses the “,” operator when it returns the XmlNamespaceManager. The XmlNamespaceManager implements IEnumerable and returning it inside of an array prevents the XmlNamespace itself from being enumerated by PowerShell when it is returned, which is what we want.

Lastly the get-XSLT_LiteralXPathNavigator function is used to make an XPathNavigator out of literal XML.

filter get-XSLT_LiteralXPathNavigator
{
param ([string]$literalXml)
[xml]$xml = $literalXml;
$xml.CreateNavigator();
}

This is a case where assuming the XML is in fact text is ok, becuase it is text and we use the PowerShell implementation of XML to get a navigator from the string.

You probably have noticed the all of these functions have an “XSLT_” internal prefix in them but this blog article doesn’t show anything about the use of XSLT. There is more comming in blog articles that follow this one…

So where are we at? The xeval function can handle a lot of the kinds of processing that is typically done with XML and has none of the limitations that the PowerShell implemenation of XML does. You do have to learn a bit about XPath, the references that were sited earlier are a good place to start and there are XPath tutorials all over the web. YMMV, but typically the best way to process XML is to process XML rather than turn it into an object model. It will be worth you effort at learning XPath… after all it is easier to learn XPath than to learn Perl:-).

PowerShell and XmlDocument

Dan Sullivan — Mon, 30 Oct 2006 15:09:00 GMT

PowerShell has builtin support for XML, but the System.Xml namespace offers many additional capabilites for processing XML. This article looks at using System.Xml in PowerShell. This article assumes you know some of the basics of PowerShell programming and are familiar with the System.Xml namespace in .NET.

First of all the [xml] data type variable in PowerShell is an instance of an XmlDocument. Typically an [xml] variable is used by assigning a variable to it. For example:

PS C:\demos> [xml]$xmldata = "<order><line price='100' qty='3'>hammer</line></order>"

We are going to look at a different way to load XML into an [xml] variable.

PS C:\demos> $xmldata = new-object "System.Xml.XmlDocument"
PS C:\demos> $xmldata.LoadXml("<order><line price='100' qty='3'>hammer</line></order>")
PS C:\demos>

This way of initializing a variable produces the same result as assigning a string to an [xml] variable, but does not create a new instance of an XmlDocument. We don’t want to create a new instance of an XmlDocument so we can leverage XPathNavigators and XPathExpressions but that will become evident later.

You can get back the XML in text form by using the get_InnerXml() method.

PS C:\demos> $xmldata.get_InnerXml()
<order><line price="100" qty="3">hammer</line></order>
PS C:\demos>

If you were programming in C# you would use the InnerXml property of the $xmldata variable to retrieve the InnerXml, but you must use the underlying get_InnerXml method in PowerShell to do the same thing. This will be the case for the other properties in XmlDocument too, the properties are not available by just using their name as the are in C#.

The root of an XML document is called the DocumentElement and for our document the name the root element is “order”. You can get a reference to it through the DocumentElement property and use its Name property to find its name.

PS C:\demos> $xmldata.get_DocumentElement().get_Name()
order
PS C:\demos>

Here we used the get_DocumentElement method to get the DocumentElement and the get_Name method to get its name.

You can modify an XmlDocument by adding or removing XML nodes. A node is part of XML, for example an element is a node as is an attribute. Let’s add another line to our order.

PS C:\demos> $line = $xmldata.CreateElement("line")
PS C:\demos> $line.SetAttribute("price", 23)
PS C:\demos> $line.SetAttribute("qty", 4)
PS C:\demos> $line.set_InnerText("nail")
PS C:\demos> $d =$xmldata.get_DocumentElement().AppendChild($line)
PS C:\demos> $xmldata.get_InnerXml()
<order><line price="100" qty="3">hammer</line><line price="23" qty="4">nail</line></order>
PS C:\demos>

Elements for an XmlDocument are not created using a constructor. Instead a technique called a factory method is used to create them. This is typical of almost all XML processors on any platform. An XmlDocument contains factory methods to create the various kinds of nodes you find in an XmlDocument. To create a new line element we use the XmlDocument.CreateElement factory method.

An XmlElement has a SetAttibute that is used to add attibutes to that element. We use the SetAttribute to add a “price” and “qty” attibute to the line element we created. SetAttribute is really a shortcut method. We could use CreateAttribute and SetAttributeNode instead, but SetAttribute is more straightforward.

The content of the new line element is set using the set_InnerText method. Again, if you were programming in C# you would assigning the InnerText property a value, but for an XmlDocument you must use the set_InnerText method. This will be true for the other assignable properties in XmlDocument too.

Creating a element using a factory method does not add that element to the document. We use the AppendChild method of the DocumentElement of the document to add the line. Append child always makes the added element the last child. PrependElement will also add an element but will make it the first child element. AppendChild always returns a reference to the element that was appended. To prevent that returned value from “leaking” out of the script we capture it in the dummy $d variable.

Last we use the get_InnerXml method of the XmlDocument see that we have in fact added a new line element to the document.

Now that we have a document let’s do some processing of it. Each line has a price and qty attribute and the product of these two attributes is called the extended price. The value of an order is the sum of all of the extended prices in it. So let’s calculate the value of an order. In this example we are going to use the native [xml] support built into PowerShell.

To start with let’s just calculate the extended prices.

PS C:\demos> $xmldata.order.line | %{$_.price * $_.qty}
100100100
23232323
PS C:\demos>

Somehow the results don’t really look correct. Data in an XML document my be untyped or strongly typed. Untyped doesn’t really mean the data doesn’t have a type, it just means that each piece of data is considered to be a string even if it looks like a number. Strongly typed XML is produced by validating an XML document against an XML Schema. In this case the types of the pieces of data are known because they are defined in the XML Schema. Sometimes a validated XML document is called the Post Schema Validation Instance or PSVI.

Our $xmldata XML document is untyped so the price and quantity are considered to be strings. When the ‘*’ operator is used with strings the string on its right is converted to an integer, or produces an error if it cannot be converted. The value of this integer is used to replicate and concatonate the string on the left of the operator. That is why we see 100 repeated three times, the value of qty for the first line is “3”.

We have to cast the price to a double to get what we want.

PS C:\demos> $xmldata.order.line | %{[double]$_.price * $_.qty}
300
92
PS C:\demos>

Now we can use the sum function example from my previous blog article, PowerShell and XML and SQL Server, to find the value of the order.

PS C:\demos> function sumOrder {
>> begin {$value = 0}
>> process { $value += [double]$_.price * $_.qty}
>> end {$value}
>> }
>>
PS C:\demos> $xmldata.order.line | sumOrder
392
PS C:\demos>

We can use the SelectNodes method of XmlDocument to get the same result. It makes use of an XPath expression which is a bit more flexible, though more complicated, than the dotted syntax that PowerShell provides for [xml] variables.

PS C:\demos> $xmldata.SelectNodes("//line") | sumOrder
392
PS C:\demos>

The SelectNodes method returns a set of XML nodes from a document that meet some criterion specified by the XPath expression. This particular XPath expression returns all the line elements in the document. When a set of XML nodes is put into a pipeline PowerShell passes each one of the nodes one at a time into the pipe.

One of the nice things about using XPath is that you can bury a lot of selection logic right into the XPath expression. What if we want to know the value of only the expensive items in our order? Our definition of expensive is when the price is more than 99.

PS C:\demos> $xmldata.SelectNodes("//line[@price>99]") | sumOrder
300
PS C:\demos>

Here our XPath expression has a predicate, “[@price>99]”, that filters out any lines whose value is 99 or less. The following is the equivalent using the XML capabilities built into PowerShell.

PS C:\demos> $xmldata.order.line | ?{[double]$_.price -gt 99} | sumOrder
300
PS C:\demos>

Notice that in this case it was important to cast the price as a [double] otherwise PowerShell would have taken the filter to check to see if the price lexically sorted after the string “99”.

An important thing about XPath is its universality. The XPath expression we have used in this example, technically called LocationPath, is a criterion for selection. Virtually every language and every platform supports XPath. You can pass the XPath expression we used in this example, “line[@price>99}”, to almost any other program and it will select the same lines for processing as we did in this example.

You have to be careful reading XML. For example below is an XML file that is encoded as big endian UTF-16. You can’t see the actual encoding on this page but you can download this test file from http://www.pluralsight.com/dan/samples/PSXml.zip if you want to try it out.

<?xml version="1.0" encoding="UTF-16BE"?>
<Test/>

The get-content command is a way to read the content of a file. For example you might try to read the sample file into a builtin [xml] datatype in PowerShell like this:

PS C:\demos> [xml]$x = get-content "c:\demos\testdocs\test.xml"
Cannot convert value "System.Object[]" to type "System.Xml.XmlDocument". Error:
 "Root element is missing."
At line:1 char:8
+ [xml]$x  <<<< = get-content "c:\demos\testdocs\test.xml"
PS C:\demos>

What PowerShell is doing here is reading in test.xml as string, then assigning that string to then $x variable. Unfortunately when it does this it has to make a guess about the encoding of the file because I didn’t tell it what it was and it guessed wrong. In fact if I just ask it to read the file and tell it I just don’t know the encoding it will generate a lot of unknown characters because of an incorrect guess about the encoding of the file.

PS C:\demos> get-content "c:\demos\testdocs\test.xml" -encoding Unknown
??????????????????????????????????????????????????????
PS C:\demos>

However I happen to know the encoding for the file, as I said it big endian UTF-16, so I can do this:

PS C:\demos> [xml]$x = get-content "c:\demos\testdocs\test.xml" -encoding BigEndianUnicode
PS C:\demos> $x.get_InnerXml()
<?xml version="1.0" encoding="UTF-16BE"?><Test />
PS C:\demos>

Now we are able to read our big endian UTF-16 file. But this defeats one of the most important features of XML; You can read an XML file without knowing the encoding.

Fortunately because PowerShell supports all of the .NET framework we can get around this problem and read any XML file that the underlying .NET Framework can handle without knowing its encoding.

PS C:\demos> $doc = new-object "System.Xml.XmlDocument"
PS C:\demos> $doc.Load($filePath)
PS C:\demos> $doc.get_InnerXml()
<?xml version="1.0" encoding="UTF-16BE"?><Test />
PS C:\demos>

Here we initialize $doc as an XmlDocument then use the Load function an XmlDocument to load in the file. The argument for Load is a string that can be either a file path or a URL. This is the recommended way to load an XML document into a variable because you shouldn’t depend on knowing what encoding of an XML document is.

Now that we can read XML lets process some XML files. Microsoft Word 2003 can be saved as XML. We have a few files that have been saved this way.

PS C:\demos> dir c:\demos\testdocs\*.xml
    Directory: Microsoft.PowerShell.Core\FileSystem::C:\demos\testdocs
Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---        10/30/2006  10:29 AM        108 betest.xml
-a---        10/30/2006   9:29 AM      24982 Test Document 1.xml
-a---        10/30/2006   9:33 AM      29195 Test Document 2.xml
-a---        10/30/2006   9:30 AM      25105 Test Document 3.xml
-a---        10/30/2006  10:05 AM        108 test.xml
PS C:\demos>

Actually some of the files in this directory are not Office documents, so we need a way to distinguish them. All Office XML files have on thing in common, they start with something called a processing instruction that looks like:

<?mso-application progid="Word.Document"?>

Let’s build a filter that will skip over the files that are not Word documents.

PS C:\demos> dir c:\demos\testdocs\*.xml | ?{$x = new-object "System.Xml.XmlDocument";
>> $x.Load($_.FullName);
>> $x.SelectSingleNode("processing-instruction('mso-application')")}
>>
    Directory: Microsoft.PowerShell.Core\FileSystem::C:\demos\testdocs
Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---        10/30/2006   4:29 PM      32448 Test Document 1.xml
-a---        10/30/2006   4:11 PM      29935 Test Document 2.xml
-a---        10/30/2006   9:30 AM      25105 Test Document 3.xml
PS C:\demos>

Here we use the full path name for each file to load an XmlDocument. Then we use the SelectSingleNode method of XmlDocument to see if we can find the processing instruction we are looking for. If the processing function isn’t found then the file name is not passed out of the pipe so it does not get listed.

The XPath expression we used was bit more complicated than the first one we tried. If you are interested in an interactive tool for working with XPath you can download Aaron Skonnard’s XPath expression builder from http://www.pluralsight.com/toolcontent/xpath-expression-builder-4.zip. Also these test documents can be found at http://www.pluralsight.com/dan/samples/PSXml.zip.

Lastly we would like to show that these Word documents have been processed by PowerShell. After you open a document in Word if you go to File->Properties->Custom you will see that you can add custom properties of you own design to a word document. We would like to add a PowerShell custom property that indicates when the document was processed by PowerShell. These properties are embeded into the XML for the Word document.

Another thing about Word documents that we haven’t you looked at is that they make heavy use of XML namespaces. So before we try anything with a complete Word document let’s look at simple document that has namespaces in it.

PS C:\demos> [xml]$x = '<w:wordDocument
>> xmlns:w="http://schemas.microsoft.com/office/word/2003/wordml"
>> xmlns:o="urn:schemas-microsoft-com:office:office">
>> <o:CustomDocumentProperties>
>> </o:CustomDocumentProperties>
>>  </w:wordDocument>'
>>
PS C:\demos> $x.get_InnerXml()
<w:wordDocument xmlns:w="http://schemas.microsoft.com/office/word/2003/wordml"
xmlns:o="urn:schemas-microsoft-com:office:office"><o:CustomDocumentProperties><
/o:CustomDocumentProperties></w:wordDocument>
PS C:\demos>

This document is a mini-Word document with all the things we don’t care about stripped out of it.

Custom properties for a Word document are contained in a CustomDocumentProperties element from the “urn:schemas-microsoft-com:office:office” namespace. If the Word document doesn’t have any custom properties it will not have this element. So we will need a way to check to see if that element in the document. Let’s test our mini-Word document to see verify we can find it.

PS C:\demos> $custDoc = "/*/*[local-name()='CustomDocumentProperties' and namespace-uri()='urn:schemas-microsoft-com:office:office']"
icrosoft-com:office:office']
PS C:\demos> $x | ?{$_.SelectSingleNode($custDoc)}
wordDocument
------------
wordDocument

Here we have made an XPath expression and saved it into a variable so we can easily reuse it. It looks for an element whose name is CustomDocumentProperties and is in the office namespace. We can use it in a simple filter test and see that our test document can get through the filter.

Next let’s look at adding the CustomDocumentProperites if it is not there. First of all we will need an element to add.

PS C:\demos> $props = $x.CreateElement("CustomDocumentProperties", $custDoc)
PS C:\demos> $props.get_OuterXml()
<CustomDocumentProperties xmlns="/*/*[local-name()='CustomDocumentProperties' and namespace-uri()='urn:schemas-microsoft-com:office:office']" />
PS C:\demos>

To create an element in a particular namespace we use the second parameter of the CreateElement method to specify the desired namespace. To check to see if we got what we wanted we use the OuterXml property… there is no InnerXml for an element with no content. Let’s make a test document without a CustomDocumentProperties and try adding this element.

PS C:\demos> $x2.get_InnerXml()
<w:wordDocument xmlns:w="http://schemas.microsoft.com/office/word/2003/wordml"/>
PS C:\demos> if ($x2.SelectSingleNode($custDoc)){}else
>> {
>> $props = $x2.CreateElement("CustomDocumentProperties", "urn:schemas-microsoft-com:office:office")
>> $x2.get_DocumentElement().AppendChild($props)
>> }
>>
PS C:\demos> $x2.get_InnerXml()
<w:wordDocument xmlns:w="http://schemas.microsoft.com/office/word/2003/wordml">
<CustomerDocumentProperties xmlns="urn:schemas-microsoft-com:office:office" /><
/w:wordDocument>
PS C:\demos>

Here we start off with a new XML document, $x2 that contains just a wordDocument. We use an if construct to test the $x2 to see if it contains CustomDocumentProperties element. If if it does not we create on and add it. Then we check to make sure the element was added.

This will be a useful for what we do next so let’s save it as a function.

function addCustomProps
{
$cust = $_.SelectSingleNode("/*/*[local-name()='CustomDocumentProperties' and namespace-uri()='urn:schemas-microsoft-com:office:office']")
if($cust){$cust}else{
$props = $_.CreateElement("CustomDocumentProperties", "urn:schemas-microsoft-com:office:office")
$_.get_DocumentElement().AppendChild($props)
}
}

Note that the addCustomProps function always returns a CustomDocumentProperties.

Now we have everthing we need to modify a Word document by adding a custom property to it.

PS C:\demos> $filePath = "C:\demos\testdocs\test document 1.xml"
PS C:\demos> $doc = new-object "System.Xml.XmlDocument"
PS C:\demos> $doc.Load($filePath)
PS C:\demos> $prop = $doc.CreateElement("PowerShell", "urn:schemas-microsoft-com
:office:office")
PS C:\demos> $prop.SetAttribute("dt", "uuid:C2F41010-65B3-11d1-A29F-00AA00C14882
", "string")
string
PS C:\demos> $prop.set_InnerText([System.DateTime]::Now)
PS C:\demos> $doc | %{addCustomProps} | %{$_.AppendChild($prop)}
dt                                      #text
--                                      -----
string                                  10/30/2006 16:07:57

PS C:\demos>
PS C:\demos> $doc.Save($filePath)

We start off by setting the $filePath variable to the path of a Word document. Next we load that Word document into the $doc variable.

We use the $doc variable to create a PowerShell element, and fill it out with the current time. We also add a dt attribute to specify that this is a string property and put the PowerShell element in the “urn:schemas-microsoft.com:office:office” namespace. Both of these are required for a custom property added to a Word document.

Finally we pass the $doc property through a pipeline our addCustomProps function. This function always returns the CustomDocumentProperty element so we can use the next segment of the pipeline to append our PowerShell property to it.

If you now open the “test document 1.xml” file in Word and navigate to its custom properties you will see that is now has a PowerShell property.

So we can use the full set of features available from the System.Xml namespace in .NET. The key to really making use of the is to become familiar with XPath. We really have just scratched the surface of its capabilities.

Dan

dan@pluralsight.com

PowerShell and XML and SQL Server

Dan Sullivan — Sat, 28 Oct 2006 14:16:00 GMT

PowerShell offers support for XML data directly though its [xml] datatype and this article is going to look at that.

First of all to make use of the builtin [xml] datatype just prefix a variable name with [xml] when you assign something to it.

PS C:\demos> [xml]$order = "<order customer='joe'>
>>         <line price='3.00' qty='4'>nut</line>
>>         <line price='1.00' qty='3'>hammer</line>
>>         </order>"
>>

At first glance this appears to be an ordinary assignment of a string to a variable. But it is doing more than that, it is checking the string to make sure that it is well-formed XML, i.e. text that conforms to the XML specification. For example the following will produce an error

PS C:\demos> [xml]$xdata = "<order ><line>stuff</line></Order>"
Cannot convert value "<order ><line>stuff</line><Order>" to type "System.Xml.Xm
lDocument". Error: "Unexpected end of file has occurred. The following elements
 are not closed: Order, order. Line 1, position 34."
At line:1 char:12
+ [xml]$xdata  <<<< = "<order ><line>stuff</line><Order>"

because of a mis-match between the opening <order> tag and the closing <\Order> tag. So when you use an [xml] variable in PowerShell you can be sure that what it contains is XML.

So $order is an [xml] variable, what can we do with it? One of the nice things about PowerShell is that if you are not sure what you can do with a varable you can always ask by piping the variable to the get-member function.

PS C:\demos> $order | get-member

   TypeName: System.Xml.XmlDocument
Name                        MemberType            Definition
----                        ----------            ----------
ToString                    CodeMethod            static System.String XmlNo...
add_NodeChanged             Method                System.Void add_NodeChange...
add_NodeChanging            Method                System.Void add_NodeChangi...
add_NodeInserted            Method                System.Void add_NodeInsert...
add_NodeInserting           Method                System.Void add_NodeInsert...
add_NodeRemoved             Method                System.Void add_NodeRemove...
add_NodeRemoving            Method                System.Void add_NodeRemovi...
AppendChild                 Method                System.Xml.XmlNode AppendC...
Clone                       Method                System.Xml.XmlNode Clone()
CloneNode                   Method                System.Xml.XmlNode CloneNo...
CreateAttribute             Method                System.Xml.XmlAttribute Cr...
CreateCDataSection          Method                System.Xml.XmlCDataSection...
CreateComment               Method                System.Xml.XmlComment Crea...
CreateDocumentFragment      Method                System.Xml.XmlDocumentFrag...
CreateDocumentType          Method                System.Xml.XmlDocumentType...
CreateElement               Method                System.Xml.XmlElement Crea...
CreateEntityReference       Method                System.Xml.XmlEntityRefere...
CreateNavigator             Method                System.Xml.XPath.XPathNavi...
CreateNode                  Method                System.Xml.XmlNode CreateN...
CreateProcessingInstruction Method                System.Xml.XmlProcessingIn...
CreateSignificantWhitespace Method                System.Xml.XmlSignificantW...
CreateTextNode              Method                System.Xml.XmlText CreateT...
CreateWhitespace            Method                System.Xml.XmlWhitespace C...
CreateXmlDeclaration        Method                System.Xml.XmlDeclaration ...
Equals                      Method                System.Boolean Equals(Obje...
GetElementById              Method                System.Xml.XmlElement GetE...
GetElementsByTagName        Method                System.Xml.XmlNodeList Get...
GetEnumerator               Method                System.Collections.IEnumer...
GetHashCode                 Method                System.Int32 GetHashCode()
GetNamespaceOfPrefix        Method                System.String GetNamespace...
GetPrefixOfNamespace        Method                System.String GetPrefixOfN...
GetType                     Method                System.Type GetType()
get_Attributes              Method                System.Xml.XmlAttributeCol...
get_BaseURI                 Method                System.String get_BaseURI()
get_ChildNodes              Method                System.Xml.XmlNodeList get...
get_DocumentElement         Method                System.Xml.XmlElement get_...
get_DocumentType            Method                System.Xml.XmlDocumentType...
get_FirstChild              Method                System.Xml.XmlNode get_Fir...
get_HasChildNodes           Method                System.Boolean get_HasChil...
get_Implementation          Method                System.Xml.XmlImplementati...
get_InnerText               Method                System.String get_InnerText()
get_InnerXml                Method                System.String get_InnerXml()
get_IsReadOnly              Method                System.Boolean get_IsReadO...
get_Item                    Method                System.Xml.XmlElement get_...
get_LastChild               Method                System.Xml.XmlNode get_Las...
get_LocalName               Method                System.String get_LocalName()
get_Name                    Method                System.String get_Name()
get_NamespaceURI            Method                System.String get_Namespac...
get_NameTable               Method                System.Xml.XmlNameTable ge...
get_NextSibling             Method                System.Xml.XmlNode get_Nex...
get_NodeType                Method                System.Xml.XmlNodeType get...
get_OuterXml                Method                System.String get_OuterXml()
get_OwnerDocument           Method                System.Xml.XmlDocument get...
get_ParentNode              Method                System.Xml.XmlNode get_Par...
get_Prefix                  Method                System.String get_Prefix()
get_PreserveWhitespace      Method                System.Boolean get_Preserv...
get_PreviousSibling         Method                System.Xml.XmlNode get_Pre...
get_SchemaInfo              Method                System.Xml.Schema.IXmlSche...
get_Schemas                 Method                System.Xml.Schema.XmlSchem...
get_Value                   Method                System.String get_Value()
ImportNode                  Method                System.Xml.XmlNode ImportN...
InsertAfter                 Method                System.Xml.XmlNode InsertA...
InsertBefore                Method                System.Xml.XmlNode InsertB...
Load                        Method                System.Void Load(Stream in...
LoadXml                     Method                System.Void LoadXml(String...
Normalize                   Method                System.Void Normalize()
PrependChild                Method                System.Xml.XmlNode Prepend...
ReadNode                    Method                System.Xml.XmlNode ReadNod...
RemoveAll                   Method                System.Void RemoveAll()
RemoveChild                 Method                System.Xml.XmlNode RemoveC...
remove_NodeChanged          Method                System.Void remove_NodeCha...
remove_NodeChanging         Method                System.Void remove_NodeCha...
remove_NodeInserted         Method                System.Void remove_NodeIns...
remove_NodeInserting        Method                System.Void remove_NodeIns...
remove_NodeRemoved          Method                System.Void remove_NodeRem...
remove_NodeRemoving         Method                System.Void remove_NodeRem...
ReplaceChild                Method                System.Xml.XmlNode Replace...
Save                        Method                System.Void Save(String fi...
SelectNodes                 Method                System.Xml.XmlNodeList Sel...
SelectSingleNode            Method                System.Xml.XmlNode SelectS...
set_InnerText               Method                System.Void set_InnerText(...
set_InnerXml                Method                System.Void set_InnerXml(S...
set_Prefix                  Method                System.Void set_Prefix(Str...
set_PreserveWhitespace      Method                System.Void set_PreserveWh...
set_Schemas                 Method                System.Void set_Schemas(Xm...
set_Value                   Method                System.Void set_Value(Stri...
set_XmlResolver             Method                System.Void set_XmlResolve...
Supports                    Method                System.Boolean Supports(St...
Validate                    Method                System.Void Validate(Valid...
WriteContentTo              Method                System.Void WriteContentTo...
WriteTo                     Method                System.Void WriteTo(XmlWri...
Item                        ParameterizedProperty System.Xml.XmlElement Item...
order                       Property              System.Xml.XmlElement orde...

PS C:\demos>

Notice that an [xml] PowerShell variable is an .NET XmlDocument. So everything you know about an XmlDocument can be applied to a PowerShell [xml] variable. In a later article we will look at how you can leverage this fact, but for now we will stick to the XML support that is directly part of PowerShell.

Look closely at the information returned by get-member… notice there is a property named “order” of type XmlElement. If you lookup XmlDocument in MSDN you will not find an order property for it, this is something that PowerShell has added.

The order property is in fact the root, or document, element for the $order document. PowerShell has a simple path language for XML that makes use of the additional properties that PowerShell adds to XML elements. This language is not XPath but still very useful. The simplest thing you can do with an XML variable is ask what it contains:

PS C:\demos> $order
order
-----
order

PS C:\demos>

It produces a table that shows the children of the $order variable. It shows that the $order variable has a single child named “order” whose value is “order”. Since we saw that order is a property of the $order variable we can append it with a “.” to that variable to see what that contains:.

PS C:\demos> $order.order
customer                                line
--------                                ----
joe                                     {line, line}

PS C:\demos>

When PowerShell displays the children of an element in at table, or list, it show both the elements and attributes of that element.

If you prefer to see the children of an [xml] variable in list form you can pipe the results to the format-list function:

PS C:\demos> $order.order | format-list

customer : joe
line     : {line, line}

Here we see that line has a number of lines as children.

We can extend this dotted property syntax further and find the value for customer:

PS C:\demos> $order.order.customer joe PS C:\demos>

We can tell that this is just a plain old value because the telltale “——-” used to separate the names of properties from their values is not there. Likewise if we had used a list form the name : is not there.

Now let’s use the dotted property syntax to see what the value of line is.

PS C:\demos> $order.order.line
price                      qty                        #text
-----                      ---                        -----
3.00                       4                          nut
4.00                       2                          hammer

Here we can see that PowerShell has output a two lines, one for each line element in the order element. price and qty are attributes of the line and the special name #text indicates that the line element contains some text.

Since there are a number of line elements we can refer to them individually using an array syntax.

PS C:\demos> $order.order.line[0]
price                      qty                        #text
-----                      ---                        -----
3.00                       4                          nut

PS C:\demos> $order.order.line[1]
price                      qty                        #text
-----                      ---                        -----
4.00                       2                          hammer

We can even ask how many line elements there are.

PS C:\demos> $order.order.line.Length
2

Under the covers $order.order.line is a .NET array and we can do with it anything we might do with a .NET array. If we want the price of the second line element…

PS C:\demos> $order.order.line[1].price
4.00

The dotted property syntax is actually a simple path language and is very similar to that used in data binding in ADO.NET, Windows Workflow, and Windows Presentation Framework. Now that we see the basics of this syntax lets see if we can do some calculations. Let’s calculate the value of the order.

PS C:\demos> $value = 0
PS C:\demos> $order.order.line | %{$value += [double]$_.qty * $_.price}
PS C:\demos> $value
20

Note that we had to cast the qty attribute to a double to force a numeric computation.

First we create a $value variable and initialize it to 0. Then we use the dotted property syntax to select the line elements in the $order variable. $order.order.line is actually a .NET array. When a .NET array is passed into a segment of a pipeline, as indicated by the | symbol, each item in the array is passed into that segment one at a time for processing.

The %{} syntax in the second segment of the pipeline says to execute what is between the braces once for each item that flows into the pipe. The $_ refers to the individual item that is flowing through that segment, in this case a line element. So as the line elements flow through the pipe the script between the braces calculates the extended price, i.e. the price times the qty, for the line element and accumulates that product into the $value element. Last we get the value of the $value element.

Next let’s do this same calculation again, but be more “pipeline” oriented.

PS C:\demos> $order.order.line | %{[double]$_.qty * $_.price} | &{begin{$v = 0}process{$v += $_}end{$v}}
20

This is making use of an in-line function, the part inside of &{}. It starts the same way the previous solution did, by passing the line elements into the pipeline. The second segment of the pipeline calculates the extended price for the line element and passes it onto the last segment.

The last segement is the in-line function and consists of three parts. The first part is inside of begin{} and is executed just once no matter how many elements pass through the pipeline and it initializes the $v variable to 0.

The second part of the function, the part inside of process{}, is executed once on each item that passes through the pipeline. It just adds each of the extended prices it receives to the $v variable.

The last part of the function, the part inside of the end{}, is also executed just once, but only after the process part is finished processing all of the items passing through the pipeline. All it does is refer to the $v so it is returned.

We can carry this even further. If we know we will be just summing up some numbers that pass though the pipeline we can change our in-line function into a named function and reuse it over and over.

PS C:\demos> function sum {
>> begin{$v = 0}
>> process{$v += $_}
>> end{$v}
>> }
>>
PS C:\demos> $order.order.line | %{[double]$_.qty * $_.price} | sum
20

So far our example has used literal XML, that is we have literally typed in the XML document we want to process. In real life you will probably have some XML files to process. There are a number of ways to read a file with PowerShell but the easiest it to use for our purposes is the ${} syntax. The file c:\demos\order.xml contains the same XML document we used to initialize $order in the previous examples.

PS C:\demos> [xml]$orders2 = ${c:\demos\order.xml}
PS C:\demos> $orders2
order
-----
order

PS C:\demos>

Note that the full path was specified.

Another source of XML data is SQL Server. It uses a FOR XML clause in a SELECT statement to return the results of a query in the form of XML. For example the following query is made against a table that contains information about last names there were filled out on United States census forms.

SELECT name, rank FROM LastNames FOR XML AUTO, ROOT('Names')

The LastNames table contains the names from the census forms along with their rank, ranked by how many forms contained that name. Now we have to somehow use PowerShell to execute a query against SQL Server. Besides its builtin functionality PowerShell can use any functionality provided by a .NET class. So we will use ADO.NET to execute the query. The tricky part, as we shall see in shortly, is how to stick the results into an [xml] PowerShell variable.

First we need a connection to SQL Server.

PS C:\demos> $conn = new-object System.Data.SqlClient.SqlConnection
PS C:\demos> $conn.ConnectionString = "server=.;integrated security=true;database=scratch"
PS C:\demos> $conn.Open()

To make the connection we use the PowerShell new-object function to create an instance of SqlConnection then fill out the connection string. Now we need a command.

PS C:\demos> $cmd = new-object System.Data.SqlClient.SqlCommand
PS C:\demos> $cmd.CommandText="SELECT name, rank FROM LastNames FOR XML AUTO,ROOT('Names')"
PS C:\demos> $cmd.Connection=$conn

Similar to making the connection we make the command by creating an instance of a SqlCommand and fill out the CommandText and Connection.

When ever a SqlCommand uses FOR XML it must be executed using ExecuteXmlReader, which returns its result in the form of an XmlReader.

PS C:\demos> $xrdr = $cmd.ExecuteXmlReader()

If were were writing a program in C# or VB.NET then at this point we would use the XmlReader to Load an XmlDocument. Wait! Remember what we found out when we used get-member to find out what we could do with an [xml] PowerShell variable? We found out that is was an XmlDocument. So we can just use the Load method on an [xml] varible load it with the results of our query. These is one problem though…

PS C:\demos> [xml]$names.Load($xrdr)
You cannot call a method on a null-valued expression.
At line:1 char:18
+ [xml]$names.Load( <<<< $xrdr)

A freshly created [xml] variable has not yet been initialized so we cannot use it to load anything. There are a couple of ways to intialize it. One is to just assign an XmlDocument to it.

PS C:\demos> [xml]$names = New-Object System.Xml.XmlDocument
PS C:\demos> [xml]$names.Load($xrdr)
PS C:\demos> $names
Names
-----
Names

Here we have use new-object to create an XmlDocument and then assign it to an [xml] variable. After we have done that the Load method can be used to read the results of our FOR XML query into the $order3 variable. The we have PowerShell to output the $name variable in tablular form. Here we can see it contains the expected ‘Names” root element.

Another way to initialize the [xml] varible is to just put some XML into it, then write over it when the Load method is used.

PS C:\demos> [xml]$names = '<a/>'
PS C:\demos> $names.Load($xrdr)
PS C:\demos> $names
Names
-----
Names

Now that we have our Names XML document let’s take a quick look inside of it.

PS C:\demos> $names.Names
LastNames
---------
{SMITH, JOHNSON, WILLIAMS, JONES...}

PS C:\demos> $names.Names.LastNames.Length
88799
PS C:\demos>

Here we can see that there are 88799 names the document and the first few are SMITH,JOHNSON and so on.

One last thing… there is a bug in the RC2 PowerShell that hopefully will be fixed by the time the RTM version is released. Elements whose name is “item” will cause problems when the table or list formatter processes them.

PS C:\demos> [xml]$stuff = '<stuff><item/><item/></stuf
PS C:\demos> $stuff
stuff
-----
stuff

PS C:\demos> $stuff.stuff
format-default : The member "Item" is already present.

You will still be able to process “item” element otherwise, it is just an issue when format-table or format-list is used.

We have seen the PowerShell has some pretty straightforward support for XML in its simple dotted path expressions. You can work with literal XML, XML from files and XML as it is returned from SQL Server in FOR XML queries. The PowerShell builtin path language that is very similar to that used in data binding which you may have already used. I didn’t cover it in this article, but PowerShell also allows you to make use of all the XML capabilities built into .NET… but that leaves me something to work on for the next article.

Dan

dan@pluralsight.com

Comparing XML in SQL Server 2005

Dan Sullivan — Fri, 01 Sep 2006 07:37:00 GMT

Comparing XML in SQL Server 2005

Xml is not text so a literal compare of two xml documents may lead to a false negative, that is it may indicate that two documents are not equal when in fact they are. For example these two xml documents are the same:

 <item x="1" y="2"/>
 
 
 <item y="2" x="1"/>

because the order of attributes in xml is not signicant. SQL Server has not "built in" way to compare xml documents and in fact makes a point about the fact that it cannot compare xml documents. This is not a deficiency of SQL Server, it is just due to the fact that xml may not be treated as ordinary text.

You can try to compare to xml documents by converting them to a string or a varbinary type, but this will lead to false negatives. For example:

 DECLARE @x1 xml
 DECLARE @x2 xml
 
 set @x1 = '<item x="1" y="2"/>'
 set @x2 = '<item y="2" x="1"/>'
 if CAST(@x1 AS VARBINARY(MAX)) = CAST(@x2 AS VARBINARY(MAX))
 PRINT 'equal'
 ELSE
 PRINT 'not equal'

This prints out 'not equal'

Things get more interesting if an xml schema is involved. For example:

 CREATE XML SCHEMA COLLECTION Test AS
 '
 <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
 targetNamespace = "urn:test">
 <xs:element name="item">
 <xs:complexType>
 <xs:attribute name="x" type="xs:int"/>
 <xs:attribute name="y" type="xs:int"/>
 </xs:complexType>
 </xs:element>
 </xs:schema>'
 
 DECLARE @x1 xml(Test)
 DECLARE @x2 xml
 
 set @x1 = '<item xmlns="urn:test" x="00" y="2"/>'
 set @x2 = '<item xmlns="urn:test" x="00" y="2"/>'
 if CAST(@x1 AS VARBINARY(MAX)) = CAST(@x2 AS VARBINARY(MAX))
 PRINT 'equal'
 ELSE
 PRINT 'not equal'

this prints out 'not equal' even though both @x1 and @x2 appear to be literally the same. This is due to the fact that SQL Server does not store literal form of the xml when it is strongly typed, it stores each of the simple types in the xml document in it's XML Schema canonical form. The conanical form of '00' for xs:int is '0'. So in this case the comparison is not equal becuase SQL Server is comparing '0' in the strongly typed @x1 to '00' in the weakly typed @x2.

In order to compare two xml documents the must both be converted to a canonical form (http://www.w3.org/TR/2001/REC-xml-c14n-20010315). You can then do a byte by byte comparison of the two canonical forms to determine if they are the same according to the XML 1.1 recomendation, but will not take into account the canonical represenation of data types from the XML Schema recomendation.

The XmlDsigC14NTransform class in the .NET Framwork can convert xml to canonical xml so that it can be compared. You can create a CLR user defined function that uses this class to do the comparison. For example:

 public partial class UserDefinedFunctions
 {
  [Microsoft.SqlServer.Server.SqlFunction]
  public static SqlBoolean CompareXML(SqlXml x1, SqlXml x2)
  {
   if (x1.IsNull || x2.IsNull)
   {
    return SqlBoolean.Null;
   }
   XmlDocument xdoc1 = new XmlDocument();
   xdoc1.Load(x1.CreateReader());
   XmlDocument xdoc2 = new XmlDocument();
   xdoc2.Load(x2.CreateReader());
   XmlDsigC14NTransform xcanonical1 = new XmlDsigC14NTransform();
   xcanonical1.LoadInput(xdoc1);
   XmlDsigC14NTransform xcanonical2 = new XmlDsigC14NTransform();
   xcanonical2.LoadInput(xdoc2);
   // this returns the conanical form
   Stream s1 = (Stream)xcanonical1.GetOutput(typeof(Stream));
   Stream s2 = (Stream)xcanonical2.GetOutput(typeof(Stream));
   // do byte by byte compare
   if (s1.Length != s2.Length)
   {
    return new SqlBoolean(false);
   }
   for (Int32 index = 0; index < s1.Length; index++)
   {
    if (s1.ReadByte() != s2.ReadByte())
    {
     return new SqlBoolean(false);
    }
   }
   return new SqlBoolean(true);
  }
 };

Then in T-SQL you can do the comparison:

 DECLARE @x1 xml
 DECLARE @x2 xml
 
 set @x1 = '<item x="1" y="2"/>'
 set @x2 = '<item y="2" x="1"/>'
 DECLARE @b bit
 set @b = dbo.CompareXml(@x1, @x2)
 IF @b = 1
 PRINT 'equal'
 ELSE
 PRINT 'not equal'

and this will print out 'equal' as expected.

Note that the assembly the contains the CompareXML function will have to be loaded with PERMISSION_SET = UNSAFE because XmlDsigC14NTransform.LoadInput may not be used from partially trusted assemblies.

If the following comparison is done:

 
 DECLARE @x1 xml(Test)
 DECLARE @x2 xml
 
 set @x1 = '<item xmlns="urn:test" x="00" y="2"/>'
 set @x2 = '<item xmlns="urn:test" x="00" y="2"/>'
 if dbo.CompareXml(@x1,@x2) = 1
 PRINT 'equal'
 ELSE
 PRINT 'not equal'

It will return 'not equal' because SQL Server does not store literal xml for a strongly typed xml variable. It stores strongly typed xml datatype in their XML Schema canonical form and this is '0' for '00' if the datatype is xs:int. Again this is expected because of the way that SQL Server treats strongly typed xml.

Dan

What cost xml, SQL Server?

Dan Sullivan — Sun, 06 Aug 2006 15:36:00 GMT

How much space does xml cost in SQL Server 2005? This question seems to come up a lot because xml has a lot of meta data in it and to top it off SQL Server saves the xml data type using UTF-16 encoding. This is, at least for xml that is mostly ascii type characters, up twice as much space as the encoding you typically see for xml, UTF-8. So if you just want to save xml in SQL Server then return it later you're better off storing as VARCHAR(MAX) or maybe VARBINARY(MAX), right? Maybe not, but that is what this article is going to look at...

Even before we look at the space issues involved I'd like to mention that I think it is really being "too clever by half" to store xml as anything other than xml. Imagine if someone gave you a CSV file that looked like:

pipe, 100
rope, 250

and and told you the first column was the name of an item and the second was the price in whole dollars, then asked you to make a table and import it into SQL Server. You could make a table that looked like:

CREATE TABLE Stuff
(
name VARCHAR(MAX),
price VARCHAR(MAX)
)

Later when people when to use the table they could issue a query like this:

SELECT name, CAST(price AS INT) FROM Stuff

Of couse if one of the files you imported had a line like:

hose, XVII

the query would get an error at runtime when it ran. The problem here was that the data was stored as something that it wasn't. Storing xml as a string or binary is just like store an INT as text. When you insert something into an xml column SQL Server makes sure that is really is xml or rejects it, so you will not have any errors if you happen to make use of some of the xml functionality SQL Server 2005 provides. Even if you don't do make any use of the xml processing functionality in SQL Server, the client that reads the xml as text you've stored will get the error. It's usually best to catch errors as early as possible, especially if the client thinks your server made the error :-).

OK, now lets look at how much space things really take up. To do this we are going to make four tables that store xml each in a different way, fill them up with a bunch of xml then see how big they are. You can down load the code for trying these experiments yourself from http://www.pluralsight.com/dan/samples/whatcostxml.zip.

-- save as binary
CREATE TABLE XmlBinSizeTest
(
data VARBINARY(MAX)
)

--save as varchar
CREATE TABLE XmlTextSizeTest
(
data VARCHAR(MAX)
)

--save save as xml with a schema
CREATE TABLE XmlSchemaSizeTest
(
data xml(XmlSizeSchema)
)

-- save as xml without a schema
CREATE TABLE XmlSizeTest
(
data xml
)

Now we want to fill these tables up with a big walop of xml. When I need some static string or xml for testing I usually wrap it in a function so I don't have to put the literal text in my SQL expressions.

CREATE
--ALTER
FUNCTION XmlSizeTestData4()
RETURNS VARCHAR(MAX)
AS
BEGIN
RETURN
'http://www.w3.org/2001/XMLSchema-instance">

guy1@adventure-works.com" EmailPromotion="0" Phone="320-555-0195" PasswordHash="1196EB7D5425B281CACDCDD2F60F52D9689D9E49" PasswordSalt="Lanmhoo=" rowguid="D366A33A-8EDE-42BD-BF79-3E7FB9713FE1" ModifiedDate="1996-07-24T00:00:00"/>

'
END

Then a little batch to fill the tables:

DECLARE @t VARCHAR(MAX)
SET @t = dbo.XmlSizeTestData4();
DECLARE @index INT
SET @index = 100
WHILE @index > 0
BEGIN
SET @index = @index - 1
INSERT INTO XmlTextSizeTest VALUES (@t);
INSERT INTO XmlSizeTest VALUES (@t);
INSERT INTO XmlSchemaSizeTest VALUES (@t);
INSERT INTO XmlBinSizeTest VALUES (CAST(@t AS VARBINARY(MAX)));
END

We can use sys.allocation_units to find out how much space the tables take up. It takes a few joins to make this work off of table names...

SELECT o.name AS table_name, au.type_desc, au.used_pages
FROM sys.allocation_units AS au
JOIN sys.partitions AS p ON au.container_id = p.partition_id
JOIN sys.objects AS o ON p.object_id = o.object_id
WHERE o.name in (N'XmlTextSizeTest',
N'XmlSizeTest',
N'XmlSchemaSizeTest',
N'XmlBinSizeTest')

In SQL Server 2005 large objects, i.e. things like xml and VARCHAR(MAX) overflow in to supplementary pages outside of the pages for the table itself. This query tells us about both the pages used in the table and the extra overflow pages. Note that DBCC CHECKTABLE won't give you the right answer, because it won't tell you about the overflow pages. Running this query produces:

table name               type_description     used pages
XmlBinSizeTest         IN_ROW_DATA     2
XmlBinSizeTest         LOB_DATA             2802
XmlTextSizeTest       IN_ROW_DATA      2
XmlTextSizeTest       LOB_DATA             2802
XmlSchemaSizeTest IN_ROW_DATA     2
XmlSchemaSizeTest LOB_DATA            4226
XmlSizeTest              IN_ROW_DATA     2
XmlSizeTest              LOB_DATA           2907

The numbers tell an interesting story. First of all no suprise, storing text as binary takes the same amount of space as storing it as VARCHAR(MAX). However storing xml without a schema takes only about 5% more space than storing it as VARCHAR(MAX) even though the xml is being stored as UTF-16, not single byte characters as VARCHAR does. How can this be?

Let's use different xml data that really exacerbates the difference between storing xml as text and storing xml as xml. Here is another function that cobbles up some xml on the fly:

CREATE
FUNCTION XmlTestData1()
RETURNS VARCHAR(MAX)
AS
BEGIN
DECLARE @t VARCHAR(MAX)
SET @t = CAST('+ REPLICATE(CAST(' ' AS VARCHAR(MAX)), 10000)
+ '>abc'
RETURN @t
END

It's basically inserting a lot of ignorable space into the the xml. A fragment of the xml it generates looks like:

Actually it has a lot more spaces than that. To try this next you should drop all the tables and recreate them, then use the batch that fill the tables, but change the source of the xml to dbo.XmlTestData1(). After you run that batch that finds the sizes you see something like:

table name               type_description     used pages
XmlBinSizeTest         IN_ROW_DATA      2
XmlBinSizeTest         LOB_DATA            252
XmlTextSizeTest       IN_ROW_DATA      2
XmlTextSizeTest       LOB_DATA            252
XmlSchemaSizeTest IN_ROW_DATA     3
XmlSchemaSizeTest LOB_DATA           0
XmlSizeTest              IN_ROW_DATA 3
XmlSizeTest              LOB_DATA           0

It looks like saving this xml as text or binary takes up about two orders of magnitude more space than just saving it as xml!

When SQL Server saves xml it need not save it as literal text, and it doen't. It doesn't have to save the the "pointies" and it doesn't have to save whitespace that doesn't matter. And most of the the time the whitespace between consecutive opening tags doesn't matter. In other words SQL Server 2005 goes out of its way to efficiently store xml.

So it not a slam dunk to figure out which way of storing xml takes up the least amount of space. The point here isn't that xml will always take about the same or less as a corresponding VARCHAR(MAX), because sometimes the VARCHAR(MAX) or VARBINARY(MAX) will take up less space. But you will probably find that the difference isn't as great as you might suspect. In any case this is a database we are talking about, you should store xml as xml.

Dan