SQL Server

PowerSMO At Work Part II

Dan Sullivan — Mon, 12 Mar 2007 10:36:00 GMT

The next article in my series on PowerShell and SMO, PowerSMO At Work Part II is up on Simple-Talk.com now.

Dan

PowerSMO articles

Dan Sullivan — Mon, 19 Feb 2007 06:22:00 GMT

I'm working on a series of articles on PowerSMO, my combination of PowerShell and SMO, for http://www.simple-talk.com. The first few are on the site now.

Some of the topics in these articles are covered in the Applied SQL Server 2005 course.

Dan

Applied PowerSMO! I

Dan Sullivan — Wed, 08 Nov 2006 20:33:00 GMT

Now that we have PowerSMO! we can start making use of it. The first example will be building a test database. Whenever I work on an new database application or write labs for a course that involves databases I need to make test database with some data in them. T-SQL is just fine for defining tables and such, but as soon as I want to fill those tables with some sample data it gets a bit tedious… I just want to directly inject some C# into my T-SQL to manipulate strings, generate random data and so on. Now with PowerSMO! I can, in effect, do just that.

This blog article assumes you have some familiarity with PowerSMO!, SQL Server, and PowerShell. The purpose of this blog article is to show how to make use of PowerSMO! to do some typical database operations.

We start by making a database.

PS C:\demos> $server = SMO_Server
PS C:\demos> $testdb = SMO_Database $server "TestDB_1"
PS C:\demos> $testdb.DatabaseOptions.RecoveryModel="Simple"
PS C:\demos> $testdb.Create()
PS C:\demos>

To start with we need a reference to an instance of SQL Server. We are using the get-SMO_Server function from PowerSMO! to get this reference. Note that short form of the function is used, dropping the “get-” prefix. This is a handy feature of PowerShell.

$testdb is the test database we are making. SMO_Database requires a reference to a server and a name. I usually have a common prefix for test database names, we will how that is useful shortly. Also I set the RecoveryMode to “Simple” so I don’t end up with a big log on database that is really a throwaway anyhow. Lastly the $testdb doesn’t really exist until Create() is called on it, that’s what makes SMO issue the appropriate T-SQL commands to the server to create the database.

Since I, and probably you too, have a standard way to create a test database we should we should capture our ad hoc script in a function so we can reuse it. Actually we are going to make two functions here, one to create a test database name, and another to make an actual test database. We will see shortly that this will make it a lot easier to maintain things over time.

function global:get-TestDatabaseName
([string]$suffix)
{
"TestDB_{0}" -f $suffix;
}
function global:new-TestDatabase
(
[Microsoft.SqlServer.Management.Smo.Server]$server,
[string]$name_suffix)
{
$name = get-TestDatabaseName $name_suffix;
$testdb = SMO_Database $server $name;
$testdb.DatabaseOptions.RecoveryModel="Simple";
$testdb.Create();
}

get-TestDatabaseName is used to create a name for a test database. All test databases have the prefix “TestDB_”. The -f operator is the PowerShell formatting operator. It replaces {0} with the first parameter that follows it and {1} with the second and so on.

The new-TestDatabase requires a Server and a string as input. The body of the function duplicates our ad hoc script but uses the get-TestDatabaseName to generate the name of the database we want to add. Let’s try it out…

PS C:\demos> new-TestDatabase $server 3
PS C:\demos> new-TestDatabase $server 4
PS C:\demos> new-TestDatabase $server 5

It’s pretty easy to add a lot of databases. In fact you will probably find that often you end up with a lot of trash databases you want to get rid of and sometimes you just want to clean out your test databases and start over. Because we have a standard prefix for out test database that fairly easy to do. To make things even easier let’s add a function that finds all of our test databases.

function global:get-TestDatabases
([Microsoft.SqlServer.Management.Smo.Server]$server)
{
$pat = get-TestDatabaseName "*";
$server.Databases | ?{$_.Name -like $pat}
}

The get-TestDatabases function makes a pattern for test database names using the get-TestDatabaseName function. It then passes the databases it finds in $server through a -like filter that uses this pattern to eliminate the database that are not test databases.

PS C:\demos> get-TestDataBases $server | %{$_.name}
TestDB_1
TestDB_2
TestDB_3
TestDB_4
TestDB_5
PS C:\demos>

Now we can see we have made a good sized population of test databases. However it is pretty easy to get rid of all of them.

PS C:\demos> TestDatabases $server | %{$_.Drop()}
PS C:\demos> TestDatabases $server | %{$_.name}
PS C:\demos>

We use the get-TestDatabases function to pipe each of the databases into a script that calls the Drop() method on each one. A quick check shows we were successful. Ok, let’s put our test database back into the server for the rest of the things we want to do.

PS C:\demos> new-TestDatabase $server 1
PS C:\demos> $testdb = $server.Databases[(TestDatabaseName 1)]
PS C:\demos> $testdb.Name
TestDB_1
PS C:\demos>

My test systems have a few Windows users, Dawn, Don, and SqlAdmin, that I use for testing. They are all ordinary Windows users with no special priveleges and the all have logins on the Sql Server instance I’m going to do testing with. We can check to see if they are there easily.

PS C:\demos> $server.logins |  %{$_.name}
Ambler
AuditLogin
BUILTIN\Administrators
Frank
Joe
MyAsmLogin
NT AUTHORITY\SYSTEM
PARSEC5\Administrator
PARSEC5\Dawn
PARSEC5\Don
PARSEC5\SqlAdmin
PARSEC5\SQLServer2005MSFTEUser$PARSEC5$MSSQLSERVER
PARSEC5\SQLServer2005MSSQLUser$PARSEC5$MSSQLSERVER
PARSEC5\SQLServer2005SQLAgentUser$PARSEC5$MSSQLSERVER
sa
PS C:\demos>

If you look about half-way down the list you will see Dawn, Don and SqlAdmin. Of course we can refine this a bit more to list only the logins we are interested in.

PS C:\demos> $server.logins | 
    ?{$_.name -like "*\Dawn" -or $_.name -like "*\Don"
    -or $_.name -like "*\SqlAdmin"} | %{$_.name}
PARSEC5\Dawn
PARSEC5\Don
PARSEC5\SqlAdmin
PS C:\demos>

Here we use the logical -or operator and the -like pattern matching operator to filter out the logins to just the standard ones we use. In fact we should make add this to a library of functions we use when we build test databases.

function global:Get-StandardTestLogins
([Microsoft.SqlServer.Management.Smo.Server]$server)
{
$server.logins | 
    ?{$_.name -like "*\Dawn" -or 
    $_.name -like "*\Don" -or
    $_.name -like "*\SqlAdmin"}
}

The Get-StandardTestLogin function is a bit different from the ad hoc script we put together; It outputs a login object instead of just a name. Note that it uses a typed parameter for input, it requries a Server as input parameter. We can still use it to get the list of names though, even though it outputs login objects.

PS C:\demos> StandardTestLogins $server | %{$_.name}
PARSEC5\Dawn
PARSEC5\Don
PARSEC5\SqlAdmin
PS C:\demos>

If it turns out our test logins are not on the system we could use PowerShell to add them. PowerShell integrates support for WMI, Window Management Instrumentation, and is a conventional way to script new users into a Windows system or enterprise. We are not going to cover those features in this blog article though.

There is one last thing we have to check for out test logins. As the name implies SqlAdmin is supposed to be in the sysadmin role for SqlServer. That’s straight forward to check.

PS C:\demos> $server.logins["PARSEC5\SqlAdmin"].IsMember("sysadmin")
True
PS C:\demos>

Looks like we are good to go for our test logins.

Now that we know that our standard test logins are there, lets add the corresponding users to our test database.

PS C:\demos> foreach ($login in StandardTestLogins $server)
>> {
>> $user = SMO_User $testdb $login.name
>> $user.login = $login.name
>> $user.Create()
>> }
>>
PS C:\demos> $testdb.users | %{$_.name}
dbo
guest
INFORMATION_SCHEMA
PARSEC5\Dawn
PARSEC5\Don
PARSEC5\SqlAdmin
sys
PS C:\demos>

We use the PowerShell foreach command to iterate through each of our test logins. We then make a new user, $user, in the $testdb with a name the same as the login name, which is a pretty typical way to add users to a database. Then we fill out the login property of $user with the corresponding login name. Last we call the Create() method on $user. A SMO user object, like just about all new objects in SMO, do not exist in the database until after the Create() method has been called on them.

We confirm that our test users were added by listing the names of the users in $testdb.

One of the nice things about having the Get-StandardTestLogins is that we can use it to create the users for our test databases. This allows us to to keep track of out standard test logins in one place, we don’t need to constantly copy the list of them everywhere we need them.

We should turn this foreach loop into a function so we can re-use for future test databases.

function global:new-StandardTestUsers
([Microsoft.SqlServer.Management.Smo.Database]$database)
{
foreach ($login in StandardTestLogins $database.Parent)
{
$user = SMO_User $database $login.name
$user.login = $login.name
$user.Create()
}
}

Here the new-StandardTestUsers functions requires that a database be passed into it. The body of the function is the same as the ad hoc script we wrote except that the reference to the server is gotten from the $database itself. Now, as you can see below, we just pass in a reference to our test database to the new-StandardTestUsers function to add all of our test users.

PS C:\demos> new-StandardTestUsers $testdb
PS C:\demos>

Now to finish out our test database we will add a table using SMO objects, then populate the table with some random data. The table will have an order number, a customer name and value column. The order number column will be the PRIMARY KEY.

First of all we need to make a table. In case you don’t remember to parameters to construct a table the get-SMO_ctors function will remind you.

PS C:\demos> SMO_ctors (SMOT_Table)
Table()
Table(Database database, String name)
Table(Database database, String name, String schema)
PS C:\demos>

We are not going to work with database schemas in this blog article, I’ll save that for a later one. We’ll use the second constructor.

PS C:\demos> $orders = SMO_Table $testdb "Orders"
PS C:\demos>

Now that we have a table we need to create the columns for it. Again get-SMOctors can be used to find out what parameters we need to pass to the get-SMOColumn function.

PS C:\demos> SMO_ctors (SMOT_column)
Column()
Column(SqlSmoObject parent, String name)
Column(SqlSmoObject parent, String name, DataType dataType)
PS C:\demos>

We can create a column, specify it name and type in one operation. We will need to make a DataType, so let’s check what the constructor options are for it.

PS C:\demos> smo_ctors (SMOT_DataType)
DataType()
DataType(SqlDataType sqlDataType)
DataType(SqlDataType sqlDataType, Int32 precisionOrMaxLength)
DataType(SqlDataType sqlDataType, Int32 precision, Int32 scale)
DataType(SqlDataType sqlDataType, String type)
DataType(SqlDataType sqlDataType, String type, String schema)
DataType(XmlSchemaCollection xmlSchemaCollection)
DataType(UserDefinedDataType userDefinedDataType)
DataType(UserDefinedType userDefinedType)

This, in turn requires us to make a SqlDataType, which is an enum. We can look to see what the possible enumerated values are for this too.

PS C:\demos> SMO_enum (SMOT_SqlDataType)
value__
None
BigInt
Binary
Bit
Char
DateTime
Decimal
Float
Image
Int
Money
NChar
NText
NVarChar
NVarCharMax
Real
SmallDateTime
SmallInt
SmallMoney
Text
Timestamp
TinyInt
UniqueIdentifier
UserDefinedDataType
UserDefinedType
VarBinary
VarBinaryMax
VarChar
VarCharMax
Variant
Xml
SysName
Numeric
PS C:\demos>

No big suprise here, it has all the SQL datatypes were are use to using in SQL Server. Now we can make some columns and add them to our $orders table.

PS C:\demos> $orders = SMO_Table $testdb "Orders"
PS C:\demos> $order_number = SMO_Column $orders "Order Number"      
    (SMO_DataType "Int")
PS C:\demos> $order_number = SMO_Column $orders "Order Number" 
    (SMO_DataType "Int")
PS C:\demos> $orders.Columns.Add($order_number)
PS C:\demos> $orders.Columns.Add($customer_name)
PS C:\demos> $value = SMO_Column $orders "Value"
    (SMO_DataType "Money")
PS C:\demos> $orders.Columns.Add($value)

First we create a table named “Orders”. A reference to the table is in variable $orders. Each column is made using the SMOColumn function. Note that the datatype for the column is defined using the SMODataType function.

Now that we have our table we need to make the $order_number column a primary key.

PS C:\demos> $pk = SMO_Index $orders "Orders__PK"
PS C:\demos> smo_enum (SMOT_IndexKeyType)
value__
None
DriPrimaryKey
DriUniqueKey
PS C:\demos> $pk.IndexKeyType=(SMOT_IndexKeyType)::DriPrimaryKey
PS C:\demos> $inx_col = SMO_IndexedColumn $pk "Order Number"
PS C:\demos> $pk.IndexedColumns.Add($inx_col)
PS C:\demos> $orders.Indexes.Add($pk)
PS C:\demos> $orders.Create()
PS C:\demos>

The index takes a little more work. First we make an SMOIndex.and name it “OrdersPK”. An index might include more than one column and those columns are specified in SMOIndexedColumn objects. We need to make an SMOIndexed column for each column in the index. This index is only a single column, however, so we only need to create a single SMOIndexedColumn. The SMOIndexedColumn is added to the index, $pk'. Once we have added the SMO_IndexedColumn to the SMO_Index we can add the index to the$orders` table.

Lastly we call the Create() method on $orders to make the table on the server.

Now we can add some data to the table. We can make use of the System.Random class from the .NET framework to generate some of the data. There aren’t any “row” objects for tables in SMO. We just use standard T-SQL to do inserts to add data to a table.

To fill out our table we need to produce some triples that consist of an order number, customer name, and a value. We have 30 customers with 1000 orders whose value ranges from 1 to 1000.00. With PowerShell we can use a pipeline to generate triples. Here is a simple example that generates 10 random triples.

PS C:\demos> (1..10) | %{"{0} : {1} : {2}" -f $_, 
    $rand.next(1,10), $rand.next()}
1 : 5 : 2119806400
2 : 6 : 1944455664
3 : 7 : 323953648
4 : 9 : 1680451911
5 : 4 : 820794382
6 : 4 : 1793724046
7 : 8 : 1206229307
8 : 5 : 1466323995
9 : 6 : 1123175179
10 : 1 : 1559943559
PS C:\demos>

The first part of the pipeline is an array, initialized with values from 1 to 10. The second part of the pipeline formats the numbers from the array along with some random numbers. Now we will use this technique generate the T-SQL insert statments. To get started let’s make a helper function that will create the insert statement from three input parameters.

PS C:\demos> function get-insert
>> ($number, $customer, $value)
>> {
>> "INSERT INTO Orders Values ({0}, '{1}', {2})" -f
>> $number, $customer, $value
>> }
>>
PS C:\demos> insert 1 "joe" 102.32
INSERT INTO Orders Values (1, 'joe', 102.32)
PS C:\demos>

The get-insert function uses the -f operator to build an insert statement from the three parameters passed into it.

PS C:\demos> (1..10) |
    %{insert $_ ("Name_" + $rand.next(1,30)) 
    ($rand.next(1,1000000)/100.1)}
INSERT INTO Orders Values (1, 'Name_15', 1127.23276723277)
INSERT INTO Orders Values (2, 'Name_12', 9753.04695304695)
INSERT INTO Orders Values (3, 'Name_23', 9560.27972027972)
INSERT INTO Orders Values (4, 'Name_15', 2843.98601398601)
INSERT INTO Orders Values (5, 'Name_13', 2798.22177822178)
INSERT INTO Orders Values (6, 'Name_13', 4531.40859140859)
INSERT INTO Orders Values (7, 'Name_11', 4825.18481518482)
INSERT INTO Orders Values (8, 'Name_26', 8539.9000999001)
INSERT INTO Orders Values (9, 'Name_22', 9863.88611388611)
INSERT INTO Orders Values (10, 'Name_10', 4556.79320679321)
PS C:\demos>

Here we have used the get-insert function generate ten insert’s. Next we need a way to execute those insert statements. A SMO database object has a number methods that can execute a SQL statement, much like a SqlCommand does in ADO.NET. In this case we will use the ExecuteNonQuery method of $database to insert 1000 orders.

PS C:\demos> (1..1000) | 
    %{insert $_ ("Name_" + $rand.next(1,30))
    ($rand.next(1,1000000)/100.1)} | 
    %{$testdb.ExecuteNonQuery($_)}
PS C:\demos>

We put an array with a thousand numbers into a pipeline, each number is used as an order number. The customer name is just the string “name_” with a suffix that is a random number from 1 to 30. And then for the value we generate a random number. These constructed values are passed into the get-insert function to make an insert statement. The insert statement is then used a the parameter to the ExectureNonQuery method on $testdb.

We can see the row count of the table is now 1000.

PS C:\demos> $orders.refresh()
PS C:\demos> $orders.rowcount
1000
PS C:\demos>

Note that before we check the rowcount we refresh the $orders table.

To finish up we will make use of the fact that can also execute a select statement against the Orders database.

PS C:\demos> $dataset = $testdb.ExecuteWithResults(
    "SELECT * FROM Orders")
PS C:\demos> $table = $dataset.tables[0]
PS C:\demos> $table.Rows | format-table -autosize
Order Number Customer Name     Value
------------ -------------     -----
           1 Name_29       1837.6124
           2 Name_27       3723.3666
           3 Name_1        3221.3886
...

The ExecuteWithResults method of $database returns an ADO Dataset. We can extract the rows from the first table in the Dataset and pipe them into a PowerShell format-table command to see what they contain.

We have seen that we can use PowerSMO! to create a test database, add tables to it and then fill these tables with some random data. Along the way we have seen how to use PowerSMO! to do a number of typical database operation like checking out what logins exist or adding users to database.

Well that’s it for this article, there will be more later.

Dan dan@pluralsight.com

PowerSMO!

Dan Sullivan — Tue, 07 Nov 2006 13:55:00 GMT

Last year I wrote a blog article about using what was then called MSH with SQL Server Management Objects http://pluralsight.com/blogs/dan/archive/2005/12/29/17703.aspx. MSH is now called PowerShell and mixing some SMO with it makes PowerSMO!

SMO is a set of object models for SQL Server. With PowerSMO! you can manipulate those object models from the command line or with a script. The two object models probably of most interest to SQL Server developers and DBA’s are the Server and ManagedComputer object models and that’s what this article is going to use them to show how to use PowerSMO!.

This article assumes you are reasonably familiar with SQL Server and have some familiarity with PowerShell. You can look at some of the previous articles I have blogged about PowerShell to get familiar with PowerShell.

There are a number of reasons for using PowerSMO! You can script the management, testing, status, etc. of database objects. PowerSMO! also has pedagogical value too. I’ve been using PowerShell + SMO in the classes I teach because it provides such a quick way too cobble up ad hoc examples. It’s useful in development for the same reason, you can quickly try something out before you commit the time to edit/compile/test a C# program.

To use PowerSMO! you need to start up PowerShell then run the InitPowerSMO.ps1 script. This script is available at http://www.pluralsight.com/dan/samples/InitPowerSMO.zip.

Windows PowerShell
Copyright (C) 2006 Microsoft Corporation. All rights reserved.
PS C:\demos> . "C:\demos\InitPowerSMO.ps1"
PS C:\demos>

This script loads the SMO assemblies into PowerShell and a number of other things that we will look at later.

Let’s start with some basics. The new-object cmd-let is used to make an instance of a .NET class. SMO has a class named ManagedComputer. An instance of ManagedComputer represents the root of one of the object models in SMO, the one that is used to manage the services provided by SQL Server, such as the database engine or full-text search. BTW, watch out for the line-wraps in the examples that follow.

PS C:\demos> $mc = new-object "Microsoft.SqlServer.Management.Smo.Wmi.ManagedComputer"
PS C:\demos> $mc.services | %{$_.name}
MSFTESQL
MSSQLSERVER
SQLBrowser
SQLSERVERAGENT
PS C:\demos>

Here we used an instance of the ManagedComputer class find the SQL Server services on the local computer. We piped these services to a script that prints out the name of the service. Looks like a pretty typical SQL Server installation.

This is handy but, as you can see, it is a bit tedious dealing with the full name of the class that new-object requires. PowerShell deals with this issue in a number of places by using what it calls a “type accelerator” or “type alias”.

For example [wmi] is a type accelerator for System.Management.ManagementObject and you can use it instead of the fully spelled out class name when you need to specify that type. Unfortunately there is no provision in PowerShell for creating your own type aliases. PowerShell supports user defined aliases for cmdlets, functions, etc. but these are not useful as type accelerators.

To save on typing PowerSMO! creates helper functions that can be used to create instances of the classes used by SMO. One of the reasons you see a delay when you run the initPowerSMO.ps1 script is that it has to create all these functions. Below is a script that uses one of these functions and duplicates the operation of the previous example.

PS C:\demos> $mc = Get-SMO_ManagedComputer
PS C:\demos> $mc.Services | %{$_.name}
MSFTESQL
MSSQLSERVER
SQLBrowser
SQLSERVERAGENT
PS C:\demos>

All of these helper functions have a preface of “Get-SMO_” followed by the name of the class that they create.

All that the Get-SMO_ helper functions and new-object are doing is running the constructor for the class requested. In the previous two samples we used the parameter-less constructor to make a ManagedComputer object. When you do this the managed computer is the one PowerShell is running on.

One of the nice things about PowerShell is that you can always ask what kinds of things an object can do by piping it to the get-member cmdlet. For example:

PS C:\demos> $mc | get-member

   TypeName: Microsoft.SqlServer.Management.Smo.Wmi.ManagedComputer
Name                   MemberType Definition
----                   ---------- ----------
Equals                 Method     System.Boolean Equals(Object obj)
GetHashCode            Method     System.Int32 GetHashCode()
GetSmoObject           Method     Microsoft.SqlServer.Management.Smo.Wmi.Wmi...
...

PowerShell does not have the same level of builtin “What can you do for me?” capability for the constructors of .NET classes. .NET classes often have a number of constructors to choose from and it is often hard to remember them all. PowerSMO! has some helper functions to make it easier to see what constructors there are for a class.

PS C:\demos> Get-SMO_ctors (Get-SMOT_ManagedComputer)
ManagedComputer()
ManagedComputer(String machineName)
ManagedComputer(String machineName, String userName, String password)
ManagedComputer(String machineName, String userName, String password, 
ProviderArchitecture providerArchitecture)
PS C:\demos>

The Get-SMO_ctors function lists the constructors for the type passed, in parentheses to it. For every Get-SMO_ function there is a Get-SMOT_ that returns its type definition. The Get-SMO_ctors function requires a type definition, not the function that makes an instance of the type. Actually the Get-SMO_ctors function will work for any .NET type definition.

PS C:\demos> Get-SMO_ctors ([System.String])
String(Char* value)
String(Char* value, Int32 startIndex, Int32 length)
String(SByte* value)
String(SByte* value, Int32 startIndex, Int32 length)
String(SByte* value, Int32 startIndex, Int32 length, Encoding enc)
String(Char[] value, Int32 startIndex, Int32 length)
String(Char[] value)
String(Char c, Int32 count)
PS C:\demos>

The example above shows using the Get-SMO_ctors function to find the various constructors for System.String class from .NET.

All of the Get-SMO_ function will accept parameters for a constructor. You pass the parameters on the command line as you would for any other PowerShell function.

PS C:\demos> $mc = Get-SMO_ManagedComputer "PARSEC5"
PS C:\demos>

In the example above $mc is attached to a computer name “PARSEC5”.

Some constructors you will want to use require multiple parameters. The ServerConnection class serves a purpose similar to that for a SqlConnection in ADO.NET

PS C:\demos> Get-SMO_ctors (Get-SMOT_ServerConnection)
ServerConnection(SqlConnectionInfo sci)
ServerConnection(SqlConnection sqlConnection)
ServerConnection()
ServerConnection(String serverInstance)
ServerConnection(String serverInstance, String userName, String password)
ServerConnection(String serverInstance, String userName, SecureString password)
PS C:\demos>

Here you can see that there is a constructor that lets you can create a ServerConnection using a SQL login. It needs the name and password for that login.

PS C:\demos> $sc = Get-SMO_ServerConnection "PARSEC5" "ambler" "Ambler"
PS C:\demos> $sc.ConnectionString
server='PARSEC5';uid='ambler';password='Ambler';
multipleactiveresultset =false
PS C:\demos>

As you can see the ServerConnection just uses the user name and password to build the connection string it will need to connect to SQL Server.

A note about the Get-SMO_ function that is a bit different than using the new-object cmdlet. The constructor parameters for the Get-SMO_ functions are separated by spaces, unlike passing constructor parameters to the new-object cmdlet which requires them to be separted by commas.

Ok, let’s use PowerSMO! to create a database. To do this we will need to make use of two classes of SMO object, a Server and a Database. We will start by looking at the constructors for each.

PS C:\demos> Get-SMO_ctors (Get-SMOT_Server)
Server(ServerConnection serverConnection)
Server(String name)
Server()
PS C:\demos> Get-SMO_ctors(Get-SMOT_Database)
Database()
Database(Server server, String name)
PS C:\demos>

We see that to construct a Server can pass in the name of the instance of SQL Server we want to use, and for the Database we will have to pass in a reference to the Server and a the name we want for the new database.

PS C:\demos> $s = Get-SMO_Server "PARSEC5"
PS C:\demos> $db = Get-SMO_Database $s "DBDemoSample"
PS C:\demos> $db.create()
PS C:\demos> $s.Databases["DBDemoSample"].Name
DBDemoSample
PS C:\demos>

Here we have made the Server object for the default SQL Server instance on machine “PARSEC5”. Next we make the Database object by passing the variable that holds the Server object into the database constructor. Just to confirm it actually worked we look at the databases collection in the server and see if we can find a database named “DBDemoSample” and then print out its name. If there was no database named “DBDemoSample” this line would not have printed out anything.

Now let’s do a little database management. Typically when a database is created its recovery model is “Full” This is going to be a development database so we want to change its recovery mode to “Simple”. We will start by confirming our suspicions about its recovery mode.

PS C:\demos> $db = $s.DataBases["DBDemoSample"]
PS C:\demos> $db.DatabaseOptions.RecoveryModel
Full
PS C:\demos>

We use $s to get a reference to the “DBDemoSample” database. From $db we can look at the recovery model in database options and see that, in fact, it is “Full”.

The value of RecoverModel is one of the enumerated values from the Microsoft.SqlServer.Management.Smo.RecoveryModel enum. You can always find the possible values of an enum by making use of .NET reflection in PowerShell.

PS C:\demos> [Microsoft.SqlServer.Management.Smo.RecoveryModel].GetFields() | %{
$_.Name}
value__
Simple
BulkLogged
Full
PS C:\demos>

Again, similar to using new-object, you must type out the entire class name to find the enumerated values of an enum. PowerSMO! provides another helper function for finding the enumerated values.

PS C:\demos> Get-SMO_enum (Get-SMOT_RecoveryModel)
value__
Simple
BulkLogged
Full

The Get-SMO_emum function finds the enumerated values of an enum. It is similar to the Get-SMO_ctors function in that it requires a class definition, not the function that makes an instance. In this example we use the Get-SMOT_RecoveryModel to get the class definition.

Using an enumerated value once you know what it is, is fairly easy. You can just set it as though is were a string value and PowerShell will convert it to the appropriate enumeration.

PS C:\demos> $db.DatabaseOptions.RecoveryModel="Simple"
PS C:\demos> $db.Alter()
PS C:\demos>

Setting a value in a field of a SMO object does not affect the database behind it until you call the Alter() function. After the Alter() function completes the database has the new recovery mode.

Note you need not worry about setting an improper enumeration value, PowerShell will check that for you, as you can see below.

PS C:\demos> $db.DatabaseOptions.RecoveryModel = "Fullx"
Exception setting "RecoveryModel": "Cannot convert value "Fullx" to type "Micro
soft.SqlServer.Management.Smo.RecoveryModel" due to invalid enumeration values.
 Specify one of the following enumeration values and try again. The possible en
umeration values are "Full, BulkLogged, Simple"."
At line:1 char:21
+ $db.DatabaseOptions.R <<<< ecoveryModel = "Fullx"
PS C:\demos>

Now you have seen PowerSMO!. It consists of functions, prefaced by Get-SMO_ that you can use to create instances of SMO objects. It also includes Get-SMOT_ functions that you can use to retrieve the type definition for a particular kind of SMO object. The Get-SMO_ctors function gives you a list of the constructors for a class and the Get-SMO_enum function give you a list of the values for an enum.

If you take a peek inside of the initPowerSMO.ps1 file, except for the Get-SMO_enum and Get-SMO_ctors functions, you won’t see any of the functions we have used in this article. That’s because the initPowerSMO.ps1 script generates these functions from the assemblies that makeup SMO. How it does this will be the topic of a later blog article.

Dan

PowerShell and ADO.NET

Dan Sullivan — Sun, 29 Oct 2006 14:48:00 GMT

Lots of times when you are working with a SQL Server all you want to do is to poke at the data. You aren’t doing any real heavy duty transactional processing, you just want to get a look at what is going on. Of course you can fire up SSMS (SQL Server Management Studio) or SqlCmd and issue some ad hoc queries. Or you can copy all the tables you are interested in to a local instance of SQL Server on your system and do the poking there. Doing the latter is often times a lot more convienient but you need a copy of SQL Server locally and then you have to copy all the data, yadda, yadda, yadda.

Actually it’s pretty common to take the “work on the data locally” approach in C# applications. These applications use ADO.NET to build a DataSet in the application, then data bind, or add a little procedural programming or otherwise poke at it. But writing, compiling and running an ad hoc application each time you want to make a little change isn’t really practical. What does make this technique practical though is PowerShell.

In this article we’re going to look at the basics of using ADO.NET inside of PowerShell. This article assumes you know the basics of using ADO.NET in C#.

We will start with the DataSet. A DataSet is just a fancy container, just a bit more fancy than a hash table. It has tables and views and relations and queries sort of like a database does. You can use a DataSet without a database, but if you are going to use it with a database then the SqlDataAdapter is a handy way to make a snapshot of some data and save it into a dataset. Let’s start by using PowerShell to get a snapshot of the data in the Customers table of the Northwind database.

First of all we will need a query that returns all the data from the Customers table.

$query = "SELECT * FROM [Customers]"

We will also need a connection string. This one assumes a local SQL Server.

$connString = "server=.;integrated security;database=northwind"

Now we will need a DataSet to copy the customers data into. PowerShell does not initialize a variable just because it is declared so we will have to make an instance of the DataSet class.

$dataset = new-object "System.Data.DataSet" "MyDataSet"
PS C:\demos> $dataset
RemotingFormat          : Xml
SchemaSerializationMode : IncludeSchema
CaseSensitive           : False
DefaultViewManager      : {System.Data.DataViewManagerListItemTypeDescriptor}
EnforceConstraints      : True
DataSetName             : MyDataSet
Namespace               :
Prefix                  :
ExtendedProperties      : {}
HasErrors               : False
IsInitialized           : True
Locale                  : en-US
Site                    :
Relations               : {}
Tables                  : {}
Container               :
DesignMode              : False

We use the new-object command to make an instance of a DataSet. The second string passed to new-object is the name we want for the DataSet. Typing the name of a variable on the command line in PowerShell returns the properties of that object, in most cases. Here we see that the dataset has a DataSetName of “MyDataSet” as we wanted and that the Tables collection is empty.

Next we need a SqlDataAdapter so we can fill our dataset.

PS C:\demos> $da = new-object "System.Data.SqlClient.SqlDataAdapter" ($query, $connString)

Here we have used the new-object command again and passed into the SqlDataAdapter constructor both the query and the connection string. Note that when want to pass more than one parameter to a constructor for a .NET class in PowerShell, you must pass all the parameters in a single array of objects. That is why $query and $connString have been passed separated by commas and enclosed in parentheses. A comma separted list enclosed in parentheses in PowerShell creates an array of objects.

We can check to see that our DataAdapter was properly constructed by looking at the text for the SelectCommand and the ConnectionString for the connection the SelectCommand uses.

PS C:\demos> $da.SelectCommand.CommandText 
SELECT * FROM [Customers]
PS C:\demos> $da.SelectCommand.Connection.ConnectionString
server=.;integrated security=true;database=northwind

Now we have everything we need to get our snapshot.

PS C:\demos> $da.Fill($dataset)
92

All we did was to call the Fill method on $da and pass in our $dataset variable. Once we have done this we can take a look at what is changed about our dataset.

PS C:\demos> $dataset

RemotingFormat          : Xml
SchemaSerializationMode : IncludeSchema
CaseSensitive           : False
DefaultViewManager      : {System.Data.DataViewManagerListItemTypeDescriptor}
EnforceConstraints      : True
DataSetName             : MyDataSet
Namespace               :
Prefix                  :
ExtendedProperties      : {}
HasErrors               : False
IsInitialized           : True
Locale                  : en-US
Site                    :
Relations               : {}
Tables                  : {}
Container               :
DesignMode              : False

Nothing has really changed. Notice that the Table container is still empty. Don’t worry about this, is just an artifact of how PowerShell treats DataTables. If we ask how many DataTables are in $dataset we will see it contains one, as we would expect.

PS C:\demos> $dataset.Tables.Count
1
PS C:\demos>

It would be nice of the Tables container listed the names of the tables in the DataSet, but it doesn’t so we will have to write a little script to see what is there.

PS C:\demos> $dataset.Tables | %{$_.TableName}
Table

The Tables property of the $dataset is a container, so if we pass it into a pipe it sends each of its tables down the pipe one at a time. The script inside of %{} is executed once for each item that passes through the pipe and $_ is the item being passed. $_.TableName gets the name of each table that passes through the pipe. Of course there is only one in this case.

As expected, the first table that a DataAdapter loads into a DataSet is named “Table”. The Tables collection acts like an array so we can access the individual tables in the Tables collection and set their names to whatever we want.

PS C:\demos> $dataset.Tables[0].TableName = "Customers"
PS C:\demos> $dataset.Tables | %{$_.TableName}
Customers

If we want to make it a bit easier to work with the Customers table we can pull out a reference to if from the DataSet.

PS C:\demos> $customers = $dataset.Tables["Customers"]
PS C:\demos> $customers.Rows.Count
92

Note that just as we would do in a C# program we can index items in the Tables collection by their name. Once we have the $customers reference to the Customers table we can begin to work on it. We tried about the most simple thing you can do with a DataTable, we looked at how many rows there are in it.

We can look to see what kinds of columns are in the table.

PS C:\demos> $customers.Columns | Format-Table -autosize -property ColumnName, DataType
ColumnName   DataType
----------   --------
CustomerID   System.String
CompanyName  System.String
ContactName  System.String
ContactTitle System.String
Address      System.String
City         System.String
Region       System.String
PostalCode   System.String
Country      System.String
Phone        System.String
Fax          System.String

We pipe the Columns from the $customers variable into the pipe. Here we use the format-table command from PowerShell and specify that we only want to see the ColumnName and DataType properties of each column. The -autosize tells PowerShell squeeze the columns as tightly together as possible.

Lets try using PowerShell to process the data. Will start by doing something a little silly, we want the ContactNames from all of the customers whose CompanyName has at least 30 characters in it.

PS C:\demos> $customers | ?{$_["CompanyName"].Length -ge 30} | %{$_["ContactName"]}
Ana Trujillo
Diego Roel
Lino Rodriguez
Helvetius Nagy

When you pass a DataTable into a pipe it passes each of the DataTable’s rows in one at a time. The script inside of ?{} is a test, if the calculation done inside of the ?{} is true, then the object that was passed into this segment of the pipe is passed to the next. So ?{} acts as a filter and it is testing each row which is represented by $_. The last segment of the pipe just returns the ContactName from the rows that made it through the filter.

Notice that we accessed the columns of the row by using an indexer and the name of the column we wanted. That is way you access individual columns of a row when you write a C# program. However PowerShell gives us an alternate way to access a column, it add properties to the row whose names are the names of the columns. The value of each one of these column properties is the value of the column it represents. The following script does the same thing as the previous but uses the column properties.

PS C:\demos> $customers | ?{$_.CompanyName.Length -ge 30} | %{$_.ContactName}
Ana Trujillo
Diego Roel
Lino Rodriguez
Helvetius Nagy

You might find this more convient. There are a couple of gotcha’s here though. One thing to keep in mind is that the scripting language in PowerShell is very simlar to C# on purpose. It’s meant to make it straightforward to move back and forth from using PowerShell to C# without constantly have to switch your mental typing context. Another reason for this similarity is that it makes it a reasonably straightforward mechanical transformation to move code from C# to PowerShell and vica versa. So if you use this shortcut you may end up unconciously typing errors in you C# code when you are working with ADO.NET… pays your money and takes your choice.

One more gotcha with this feature is that sometimes column names in a database table have spaces in them and these have to be handled a bit differently when you use this PowerShell property short cut. Here is a simple table definition for a table in the Scratch database.

Create table TestPSData
(
[my id] INT
)

Notice that is has a single column named “my id”, which has a space in it. We can quickly fix up our data adapter to access this table and add it to our dataset.

PS C:\demos> $da.SelectCommand.CommandText="SELECT * FROM Scratch..TestPSData"
PS C:\demos> $da.Fill($dataset)
2

We can print out the ID’s in the TestPSData table using the ADO.NET indexer method.

PS C:\demos> $dataset.Tables[1] | %{$_["my id"]}
1
1

As you can see it printed out the “my id” column value for the two rows in the table.

If we try to use the PowerShell column property method we may get an error…

PS C:\demos> $dataset.Tables[1] | %{$_.my id}
Unexpected token 'id' in expression or statement.
At line:1 char:32
+ $dataset.Tables[1] | %{$_.my id} <<<<

because the space in the name makes PowerShell read an extra value that it doesn’t know how to handle. But PowerShell is a scriping language so this is easy to fix.

PS C:\demos> $dataset.Tables[1] | %{$_."my id"}
1
1

Just put some quotes around the property name so PowerShell treats it as a single word.

Columns names in SQL Server my also begin with a “$” character and this requires a bit more effort to handle, even when you are using the ADO indexer to get the column. Here is another table definition

Create table TestPSData2
(
[$myid] INT
)

Assuming we have filled out dataset up with this table instead of the previous TestPSData table then an attempt to access the $myid column can result in an error in a number of ways.

PS C:\demos> $ds2.Tables[0] | %{$_."$myid"}
PS C:\demos>

Notice that this did not produce any results but it should have produced two rows. That is becuase $myid is interpreted by PowerShell as a variable which has no value, so it is the same as using [“”]

If we try to use indexer then we get a hard error.

PS C:\demos> $ds2.Tables[0] | %{$_["$myid"]}
Unable to index into an object of type System.Data.DataRow.
At line:1 char:23
+ $ds2.Tables[0] | %{$_[" <<<< $myid"]}
Unable to index into an object of type System.Data.DataRow.
At line:1 char:23
+ $ds2.Tables[0] | %{$_[" <<<< $myid"]}
PS C:\demos>

That is because there is no column named “”.

The way to handle this problem in an both cases is to escape the $ with a backtick and enclose the whole thing in quotes.

PS C:\demos> $ds2.Tables[0] | %{$_["`$myid"]}
1
2
PS C:\demos> $ds2.Tables[0] | %{$_."`$myid"}
1
2
PS C:\demos>

OK, enough property name trivia for now.

DataTable supports a SQL-like syntax for queries. We can use that feature from PowerShell to redo are “long company name” query from before.

PS C:\demos> $customers.Select("len(CompanyName) > 30") | %{$_["ContactName"]}
Ana Trujillo
Diego Roel
Helvetius Nagy

The Select method of a Table returns a rowset. When you put a rowset into a pipe, each row is passed down the pipe as it is for a Table. In this example we, as we did before, pull the ContactNames from the rows that made it through the ADO Select filter.

We can use a rowset in PowerShell the same way we would in any other language that supports ADO.NET. For example we can use it to fill a DataTable.

Let’s start by building a DataTable from scratch.

PS C:\demos> $datatable = new-object "System.Data.DataTable"
PS C:\demos> $datatable
PS C:\demos>

It looks like we didn’t build a DataTable, but it just that PowerShell is giving us the rows of the table instead its properties and there are not yet any rows in this datatable. Let’s proceed as though there is a table and make a column for it. In fact lets make this datatable the same a the TestPSData table we used previously.

PS C:\demos> $column = new-object "System.Data.DataColumn" ("my id", [Int32])
PS C:\demos> $datatable.Columns.Add($column)
PS C:\demos> $datatable.Columns | format-table -autosize -property ColumnName, DataType
ColumnName DataType
---------- --------
my id      System.Int32

We started by making a DataColumn whose name is “my id” with datatype Int32. When you need to specify a type in PowerShell [Int32] is the equivalent of typeof(Int32) in C#.

We added the column to the $datatable columns collection then, as we did previously for the Customers table, check to see what columns the table has. Next let’s fill this datatable with the results captured in a DataReader.

The typical way we get a DataReader is to use ExecuteReader method on a SqlCommand. So lets build and execute a SqlCommand. First we will need a connection. We can use the one that is already in our DataAdapter.

PS C:\demos> $conn = $da.SelectCommand.Connection
PS C:\demos> $conn

StatisticsEnabled                : False
ConnectionString                 : server=.;integrated security=true;database=northwind
ConnectionTimeout                : 15
Database                         : northwind
DataSource                       : .
PacketSize                       : 8000
ServerVersion                    :
WorkstationId                    : PARSEC5
FireInfoMessageEventOnUserErrors : False
State                            : Closed
Site                             :
Container                        :

All we did here was to pull a reference to the SqlConnection that our DataAdapter was using. Note that the connection is closed.

Next we need a command to execute. That easy once we have a connection.

PS C:\demos> $cmd = New-Object "System.Data.SqlClient.SqlCommand" ("SELECT * FROM Scratch..TestPSData", $conn)
PS C:\demos> $cmd

Connection             : System.Data.SqlClient.SqlConnection
NotificationAutoEnlist : True
Notification           :
Transaction            :
CommandText            : SELECT * FROM Scratch..TestPSData
CommandTimeout         : 30
CommandType            : Text
DesignTimeVisible      : True
Parameters             : {}
UpdatedRowSource       : Both
Site                   :
Container              :

Now we need to execute the command and capture the results. Of course we want to be sure that the $conn is open before we use it.

PS C:\demos> $conn.Open()
PS C:\demos> $rdr = $cmd.ExecuteReader()
PS C:\demos>

What can we do with a reader? If we were writing a C# program we would probably put $rdr in a loop and call the Read() method on it until we got to the last row. We could do that in PowerShell too if we wanted to, but there is an alternate way which is more “pipeline” oriented.

PS C:\demos> $rdr | &{
>> begin{$values = new-object "System.Object[]" $rdr.FieldCount}
>> process {$_.GetValues($values); $datatable.Rows.Add($values)}
>> }
>>
1
                                                                          my id
                                                                          -----
                                                                              1
1
                                                                              1
PS C:\demos> $datatable | format-table -autosize
my id
-----
    1
    1

We start by putting the DataReader we got in previous step into the pipeline. This causes a DbDataRecord to be sent down the pipeline once for each row in the result. We process the DbDataRecords in an inline function, that is the script inside of &{}. The begin part of the function runs just once before anything is passed into the segement of the pipeline it occupies. It uses the FieldCount of the DataReader to initialize $values variable to an array that can hold all of the values of the columns in a row.

Once the begin part of the inline function has completed the process part is executed once for each DbDataRecord. It uses the GetValues method of the DbDataRecord to extract all of the values of the row and put them into the $values variable. Then it uses the Add method of the Rows property of the $datatable to add a row to the table.

Lastly we send the $datatable variable to format-table and see the the two rows from the TestPSData table in the database have been added to it.

Note that there is one artifact between the filling of the $datatable and the final command… we see the content of the table and some other stuff. This is due to the fact that the GetValues and Rows.Add methods both return values and we did not capture so they “leaked” out of the script. We can fix that just by adding a dummy variable to capture that output.

PS C:\demos> $rdr | &{
>> begin{$values = new-object "System.Object[]" $rdr.FieldCount}
>> process {$d=$_.GetValues($values); $d=$datatable.Rows.Add($values)}
>> }
>>
PS C:\demos>

So we have seen that we can use ADO.NET inside of PowerShell. It a quick way to work out some ad hoc queries and do some processing on the results. You can easily do ad hocSQL queries form SQL Server management Studio or SqlCmd, but with PowerShell you can mix the results of those queries with either pipeline or conventional procedural programming.

Dan dan@pluralsight.com

PS Thanks to Jeffery Stover for pointing out the column properties on DataTables.

PowerShell and XML and SQL Server

Dan Sullivan — Sat, 28 Oct 2006 14:16:00 GMT

PowerShell offers support for XML data directly though its [xml] datatype and this article is going to look at that.

First of all to make use of the builtin [xml] datatype just prefix a variable name with [xml] when you assign something to it.

PS C:\demos> [xml]$order = "<order customer='joe'>
>>         <line price='3.00' qty='4'>nut</line>
>>         <line price='1.00' qty='3'>hammer</line>
>>         </order>"
>>

At first glance this appears to be an ordinary assignment of a string to a variable. But it is doing more than that, it is checking the string to make sure that it is well-formed XML, i.e. text that conforms to the XML specification. For example the following will produce an error

PS C:\demos> [xml]$xdata = "<order ><line>stuff</line></Order>"
Cannot convert value "<order ><line>stuff</line><Order>" to type "System.Xml.Xm
lDocument". Error: "Unexpected end of file has occurred. The following elements
 are not closed: Order, order. Line 1, position 34."
At line:1 char:12
+ [xml]$xdata  <<<< = "<order ><line>stuff</line><Order>"

because of a mis-match between the opening <order> tag and the closing <\Order> tag. So when you use an [xml] variable in PowerShell you can be sure that what it contains is XML.

So $order is an [xml] variable, what can we do with it? One of the nice things about PowerShell is that if you are not sure what you can do with a varable you can always ask by piping the variable to the get-member function.

PS C:\demos> $order | get-member

   TypeName: System.Xml.XmlDocument
Name                        MemberType            Definition
----                        ----------            ----------
ToString                    CodeMethod            static System.String XmlNo...
add_NodeChanged             Method                System.Void add_NodeChange...
add_NodeChanging            Method                System.Void add_NodeChangi...
add_NodeInserted            Method                System.Void add_NodeInsert...
add_NodeInserting           Method                System.Void add_NodeInsert...
add_NodeRemoved             Method                System.Void add_NodeRemove...
add_NodeRemoving            Method                System.Void add_NodeRemovi...
AppendChild                 Method                System.Xml.XmlNode AppendC...
Clone                       Method                System.Xml.XmlNode Clone()
CloneNode                   Method                System.Xml.XmlNode CloneNo...
CreateAttribute             Method                System.Xml.XmlAttribute Cr...
CreateCDataSection          Method                System.Xml.XmlCDataSection...
CreateComment               Method                System.Xml.XmlComment Crea...
CreateDocumentFragment      Method                System.Xml.XmlDocumentFrag...
CreateDocumentType          Method                System.Xml.XmlDocumentType...
CreateElement               Method                System.Xml.XmlElement Crea...
CreateEntityReference       Method                System.Xml.XmlEntityRefere...
CreateNavigator             Method                System.Xml.XPath.XPathNavi...
CreateNode                  Method                System.Xml.XmlNode CreateN...
CreateProcessingInstruction Method                System.Xml.XmlProcessingIn...
CreateSignificantWhitespace Method                System.Xml.XmlSignificantW...
CreateTextNode              Method                System.Xml.XmlText CreateT...
CreateWhitespace            Method                System.Xml.XmlWhitespace C...
CreateXmlDeclaration        Method                System.Xml.XmlDeclaration ...
Equals                      Method                System.Boolean Equals(Obje...
GetElementById              Method                System.Xml.XmlElement GetE...
GetElementsByTagName        Method                System.Xml.XmlNodeList Get...
GetEnumerator               Method                System.Collections.IEnumer...
GetHashCode                 Method                System.Int32 GetHashCode()
GetNamespaceOfPrefix        Method                System.String GetNamespace...
GetPrefixOfNamespace        Method                System.String GetPrefixOfN...
GetType                     Method                System.Type GetType()
get_Attributes              Method                System.Xml.XmlAttributeCol...
get_BaseURI                 Method                System.String get_BaseURI()
get_ChildNodes              Method                System.Xml.XmlNodeList get...
get_DocumentElement         Method                System.Xml.XmlElement get_...
get_DocumentType            Method                System.Xml.XmlDocumentType...
get_FirstChild              Method                System.Xml.XmlNode get_Fir...
get_HasChildNodes           Method                System.Boolean get_HasChil...
get_Implementation          Method                System.Xml.XmlImplementati...
get_InnerText               Method                System.String get_InnerText()
get_InnerXml                Method                System.String get_InnerXml()
get_IsReadOnly              Method                System.Boolean get_IsReadO...
get_Item                    Method                System.Xml.XmlElement get_...
get_LastChild               Method                System.Xml.XmlNode get_Las...
get_LocalName               Method                System.String get_LocalName()
get_Name                    Method                System.String get_Name()
get_NamespaceURI            Method                System.String get_Namespac...
get_NameTable               Method                System.Xml.XmlNameTable ge...
get_NextSibling             Method                System.Xml.XmlNode get_Nex...
get_NodeType                Method                System.Xml.XmlNodeType get...
get_OuterXml                Method                System.String get_OuterXml()
get_OwnerDocument           Method                System.Xml.XmlDocument get...
get_ParentNode              Method                System.Xml.XmlNode get_Par...
get_Prefix                  Method                System.String get_Prefix()
get_PreserveWhitespace      Method                System.Boolean get_Preserv...
get_PreviousSibling         Method                System.Xml.XmlNode get_Pre...
get_SchemaInfo              Method                System.Xml.Schema.IXmlSche...
get_Schemas                 Method                System.Xml.Schema.XmlSchem...
get_Value                   Method                System.String get_Value()
ImportNode                  Method                System.Xml.XmlNode ImportN...
InsertAfter                 Method                System.Xml.XmlNode InsertA...
InsertBefore                Method                System.Xml.XmlNode InsertB...
Load                        Method                System.Void Load(Stream in...
LoadXml                     Method                System.Void LoadXml(String...
Normalize                   Method                System.Void Normalize()
PrependChild                Method                System.Xml.XmlNode Prepend...
ReadNode                    Method                System.Xml.XmlNode ReadNod...
RemoveAll                   Method                System.Void RemoveAll()
RemoveChild                 Method                System.Xml.XmlNode RemoveC...
remove_NodeChanged          Method                System.Void remove_NodeCha...
remove_NodeChanging         Method                System.Void remove_NodeCha...
remove_NodeInserted         Method                System.Void remove_NodeIns...
remove_NodeInserting        Method                System.Void remove_NodeIns...
remove_NodeRemoved          Method                System.Void remove_NodeRem...
remove_NodeRemoving         Method                System.Void remove_NodeRem...
ReplaceChild                Method                System.Xml.XmlNode Replace...
Save                        Method                System.Void Save(String fi...
SelectNodes                 Method                System.Xml.XmlNodeList Sel...
SelectSingleNode            Method                System.Xml.XmlNode SelectS...
set_InnerText               Method                System.Void set_InnerText(...
set_InnerXml                Method                System.Void set_InnerXml(S...
set_Prefix                  Method                System.Void set_Prefix(Str...
set_PreserveWhitespace      Method                System.Void set_PreserveWh...
set_Schemas                 Method                System.Void set_Schemas(Xm...
set_Value                   Method                System.Void set_Value(Stri...
set_XmlResolver             Method                System.Void set_XmlResolve...
Supports                    Method                System.Boolean Supports(St...
Validate                    Method                System.Void Validate(Valid...
WriteContentTo              Method                System.Void WriteContentTo...
WriteTo                     Method                System.Void WriteTo(XmlWri...
Item                        ParameterizedProperty System.Xml.XmlElement Item...
order                       Property              System.Xml.XmlElement orde...

PS C:\demos>

Notice that an [xml] PowerShell variable is an .NET XmlDocument. So everything you know about an XmlDocument can be applied to a PowerShell [xml] variable. In a later article we will look at how you can leverage this fact, but for now we will stick to the XML support that is directly part of PowerShell.

Look closely at the information returned by get-member… notice there is a property named “order” of type XmlElement. If you lookup XmlDocument in MSDN you will not find an order property for it, this is something that PowerShell has added.

The order property is in fact the root, or document, element for the $order document. PowerShell has a simple path language for XML that makes use of the additional properties that PowerShell adds to XML elements. This language is not XPath but still very useful. The simplest thing you can do with an XML variable is ask what it contains:

PS C:\demos> $order
order
-----
order

PS C:\demos>

It produces a table that shows the children of the $order variable. It shows that the $order variable has a single child named “order” whose value is “order”. Since we saw that order is a property of the $order variable we can append it with a “.” to that variable to see what that contains:.

PS C:\demos> $order.order
customer                                line
--------                                ----
joe                                     {line, line}

PS C:\demos>

When PowerShell displays the children of an element in at table, or list, it show both the elements and attributes of that element.

If you prefer to see the children of an [xml] variable in list form you can pipe the results to the format-list function:

PS C:\demos> $order.order | format-list

customer : joe
line     : {line, line}

Here we see that line has a number of lines as children.

We can extend this dotted property syntax further and find the value for customer:

PS C:\demos> $order.order.customer joe PS C:\demos>

We can tell that this is just a plain old value because the telltale “——-” used to separate the names of properties from their values is not there. Likewise if we had used a list form the name : is not there.

Now let’s use the dotted property syntax to see what the value of line is.

PS C:\demos> $order.order.line
price                      qty                        #text
-----                      ---                        -----
3.00                       4                          nut
4.00                       2                          hammer

Here we can see that PowerShell has output a two lines, one for each line element in the order element. price and qty are attributes of the line and the special name #text indicates that the line element contains some text.

Since there are a number of line elements we can refer to them individually using an array syntax.

PS C:\demos> $order.order.line[0]
price                      qty                        #text
-----                      ---                        -----
3.00                       4                          nut

PS C:\demos> $order.order.line[1]
price                      qty                        #text
-----                      ---                        -----
4.00                       2                          hammer

We can even ask how many line elements there are.

PS C:\demos> $order.order.line.Length
2

Under the covers $order.order.line is a .NET array and we can do with it anything we might do with a .NET array. If we want the price of the second line element…

PS C:\demos> $order.order.line[1].price
4.00

The dotted property syntax is actually a simple path language and is very similar to that used in data binding in ADO.NET, Windows Workflow, and Windows Presentation Framework. Now that we see the basics of this syntax lets see if we can do some calculations. Let’s calculate the value of the order.

PS C:\demos> $value = 0
PS C:\demos> $order.order.line | %{$value += [double]$_.qty * $_.price}
PS C:\demos> $value
20

Note that we had to cast the qty attribute to a double to force a numeric computation.

First we create a $value variable and initialize it to 0. Then we use the dotted property syntax to select the line elements in the $order variable. $order.order.line is actually a .NET array. When a .NET array is passed into a segment of a pipeline, as indicated by the | symbol, each item in the array is passed into that segment one at a time for processing.

The %{} syntax in the second segment of the pipeline says to execute what is between the braces once for each item that flows into the pipe. The $_ refers to the individual item that is flowing through that segment, in this case a line element. So as the line elements flow through the pipe the script between the braces calculates the extended price, i.e. the price times the qty, for the line element and accumulates that product into the $value element. Last we get the value of the $value element.

Next let’s do this same calculation again, but be more “pipeline” oriented.

PS C:\demos> $order.order.line | %{[double]$_.qty * $_.price} | &{begin{$v = 0}process{$v += $_}end{$v}}
20

This is making use of an in-line function, the part inside of &{}. It starts the same way the previous solution did, by passing the line elements into the pipeline. The second segment of the pipeline calculates the extended price for the line element and passes it onto the last segment.

The last segement is the in-line function and consists of three parts. The first part is inside of begin{} and is executed just once no matter how many elements pass through the pipeline and it initializes the $v variable to 0.

The second part of the function, the part inside of process{}, is executed once on each item that passes through the pipeline. It just adds each of the extended prices it receives to the $v variable.

The last part of the function, the part inside of the end{}, is also executed just once, but only after the process part is finished processing all of the items passing through the pipeline. All it does is refer to the $v so it is returned.

We can carry this even further. If we know we will be just summing up some numbers that pass though the pipeline we can change our in-line function into a named function and reuse it over and over.

PS C:\demos> function sum {
>> begin{$v = 0}
>> process{$v += $_}
>> end{$v}
>> }
>>
PS C:\demos> $order.order.line | %{[double]$_.qty * $_.price} | sum
20

So far our example has used literal XML, that is we have literally typed in the XML document we want to process. In real life you will probably have some XML files to process. There are a number of ways to read a file with PowerShell but the easiest it to use for our purposes is the ${} syntax. The file c:\demos\order.xml contains the same XML document we used to initialize $order in the previous examples.

PS C:\demos> [xml]$orders2 = ${c:\demos\order.xml}
PS C:\demos> $orders2
order
-----
order

PS C:\demos>

Note that the full path was specified.

Another source of XML data is SQL Server. It uses a FOR XML clause in a SELECT statement to return the results of a query in the form of XML. For example the following query is made against a table that contains information about last names there were filled out on United States census forms.

SELECT name, rank FROM LastNames FOR XML AUTO, ROOT('Names')

The LastNames table contains the names from the census forms along with their rank, ranked by how many forms contained that name. Now we have to somehow use PowerShell to execute a query against SQL Server. Besides its builtin functionality PowerShell can use any functionality provided by a .NET class. So we will use ADO.NET to execute the query. The tricky part, as we shall see in shortly, is how to stick the results into an [xml] PowerShell variable.

First we need a connection to SQL Server.

PS C:\demos> $conn = new-object System.Data.SqlClient.SqlConnection
PS C:\demos> $conn.ConnectionString = "server=.;integrated security=true;database=scratch"
PS C:\demos> $conn.Open()

To make the connection we use the PowerShell new-object function to create an instance of SqlConnection then fill out the connection string. Now we need a command.

PS C:\demos> $cmd = new-object System.Data.SqlClient.SqlCommand
PS C:\demos> $cmd.CommandText="SELECT name, rank FROM LastNames FOR XML AUTO,ROOT('Names')"
PS C:\demos> $cmd.Connection=$conn

Similar to making the connection we make the command by creating an instance of a SqlCommand and fill out the CommandText and Connection.

When ever a SqlCommand uses FOR XML it must be executed using ExecuteXmlReader, which returns its result in the form of an XmlReader.

PS C:\demos> $xrdr = $cmd.ExecuteXmlReader()

If were were writing a program in C# or VB.NET then at this point we would use the XmlReader to Load an XmlDocument. Wait! Remember what we found out when we used get-member to find out what we could do with an [xml] PowerShell variable? We found out that is was an XmlDocument. So we can just use the Load method on an [xml] varible load it with the results of our query. These is one problem though…

PS C:\demos> [xml]$names.Load($xrdr)
You cannot call a method on a null-valued expression.
At line:1 char:18
+ [xml]$names.Load( <<<< $xrdr)

A freshly created [xml] variable has not yet been initialized so we cannot use it to load anything. There are a couple of ways to intialize it. One is to just assign an XmlDocument to it.

PS C:\demos> [xml]$names = New-Object System.Xml.XmlDocument
PS C:\demos> [xml]$names.Load($xrdr)
PS C:\demos> $names
Names
-----
Names

Here we have use new-object to create an XmlDocument and then assign it to an [xml] variable. After we have done that the Load method can be used to read the results of our FOR XML query into the $order3 variable. The we have PowerShell to output the $name variable in tablular form. Here we can see it contains the expected ‘Names” root element.

Another way to initialize the [xml] varible is to just put some XML into it, then write over it when the Load method is used.

PS C:\demos> [xml]$names = '<a/>'
PS C:\demos> $names.Load($xrdr)
PS C:\demos> $names
Names
-----
Names

Now that we have our Names XML document let’s take a quick look inside of it.

PS C:\demos> $names.Names
LastNames
---------
{SMITH, JOHNSON, WILLIAMS, JONES...}

PS C:\demos> $names.Names.LastNames.Length
88799
PS C:\demos>

Here we can see that there are 88799 names the document and the first few are SMITH,JOHNSON and so on.

One last thing… there is a bug in the RC2 PowerShell that hopefully will be fixed by the time the RTM version is released. Elements whose name is “item” will cause problems when the table or list formatter processes them.

PS C:\demos> [xml]$stuff = '<stuff><item/><item/></stuf
PS C:\demos> $stuff
stuff
-----
stuff

PS C:\demos> $stuff.stuff
format-default : The member "Item" is already present.

You will still be able to process “item” element otherwise, it is just an issue when format-table or format-list is used.

We have seen the PowerShell has some pretty straightforward support for XML in its simple dotted path expressions. You can work with literal XML, XML from files and XML as it is returned from SQL Server in FOR XML queries. The PowerShell builtin path language that is very similar to that used in data binding which you may have already used. I didn’t cover it in this article, but PowerShell also allows you to make use of all the XML capabilities built into .NET… but that leaves me something to work on for the next article.

Dan

dan@pluralsight.com

Comparing XML in SQL Server 2005

Dan Sullivan — Fri, 01 Sep 2006 07:37:00 GMT

Comparing XML in SQL Server 2005

Xml is not text so a literal compare of two xml documents may lead to a false negative, that is it may indicate that two documents are not equal when in fact they are. For example these two xml documents are the same:

 <item x="1" y="2"/>
 
 
 <item y="2" x="1"/>

because the order of attributes in xml is not signicant. SQL Server has not "built in" way to compare xml documents and in fact makes a point about the fact that it cannot compare xml documents. This is not a deficiency of SQL Server, it is just due to the fact that xml may not be treated as ordinary text.

You can try to compare to xml documents by converting them to a string or a varbinary type, but this will lead to false negatives. For example:

 DECLARE @x1 xml
 DECLARE @x2 xml
 
 set @x1 = '<item x="1" y="2"/>'
 set @x2 = '<item y="2" x="1"/>'
 if CAST(@x1 AS VARBINARY(MAX)) = CAST(@x2 AS VARBINARY(MAX))
 PRINT 'equal'
 ELSE
 PRINT 'not equal'

This prints out 'not equal'

Things get more interesting if an xml schema is involved. For example:

 CREATE XML SCHEMA COLLECTION Test AS
 '
 <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
 targetNamespace = "urn:test">
 <xs:element name="item">
 <xs:complexType>
 <xs:attribute name="x" type="xs:int"/>
 <xs:attribute name="y" type="xs:int"/>
 </xs:complexType>
 </xs:element>
 </xs:schema>'
 
 DECLARE @x1 xml(Test)
 DECLARE @x2 xml
 
 set @x1 = '<item xmlns="urn:test" x="00" y="2"/>'
 set @x2 = '<item xmlns="urn:test" x="00" y="2"/>'
 if CAST(@x1 AS VARBINARY(MAX)) = CAST(@x2 AS VARBINARY(MAX))
 PRINT 'equal'
 ELSE
 PRINT 'not equal'

this prints out 'not equal' even though both @x1 and @x2 appear to be literally the same. This is due to the fact that SQL Server does not store literal form of the xml when it is strongly typed, it stores each of the simple types in the xml document in it's XML Schema canonical form. The conanical form of '00' for xs:int is '0'. So in this case the comparison is not equal becuase SQL Server is comparing '0' in the strongly typed @x1 to '00' in the weakly typed @x2.

In order to compare two xml documents the must both be converted to a canonical form (http://www.w3.org/TR/2001/REC-xml-c14n-20010315). You can then do a byte by byte comparison of the two canonical forms to determine if they are the same according to the XML 1.1 recomendation, but will not take into account the canonical represenation of data types from the XML Schema recomendation.

The XmlDsigC14NTransform class in the .NET Framwork can convert xml to canonical xml so that it can be compared. You can create a CLR user defined function that uses this class to do the comparison. For example:

 public partial class UserDefinedFunctions
 {
  [Microsoft.SqlServer.Server.SqlFunction]
  public static SqlBoolean CompareXML(SqlXml x1, SqlXml x2)
  {
   if (x1.IsNull || x2.IsNull)
   {
    return SqlBoolean.Null;
   }
   XmlDocument xdoc1 = new XmlDocument();
   xdoc1.Load(x1.CreateReader());
   XmlDocument xdoc2 = new XmlDocument();
   xdoc2.Load(x2.CreateReader());
   XmlDsigC14NTransform xcanonical1 = new XmlDsigC14NTransform();
   xcanonical1.LoadInput(xdoc1);
   XmlDsigC14NTransform xcanonical2 = new XmlDsigC14NTransform();
   xcanonical2.LoadInput(xdoc2);
   // this returns the conanical form
   Stream s1 = (Stream)xcanonical1.GetOutput(typeof(Stream));
   Stream s2 = (Stream)xcanonical2.GetOutput(typeof(Stream));
   // do byte by byte compare
   if (s1.Length != s2.Length)
   {
    return new SqlBoolean(false);
   }
   for (Int32 index = 0; index < s1.Length; index++)
   {
    if (s1.ReadByte() != s2.ReadByte())
    {
     return new SqlBoolean(false);
    }
   }
   return new SqlBoolean(true);
  }
 };

Then in T-SQL you can do the comparison:

 DECLARE @x1 xml
 DECLARE @x2 xml
 
 set @x1 = '<item x="1" y="2"/>'
 set @x2 = '<item y="2" x="1"/>'
 DECLARE @b bit
 set @b = dbo.CompareXml(@x1, @x2)
 IF @b = 1
 PRINT 'equal'
 ELSE
 PRINT 'not equal'

and this will print out 'equal' as expected.

Note that the assembly the contains the CompareXML function will have to be loaded with PERMISSION_SET = UNSAFE because XmlDsigC14NTransform.LoadInput may not be used from partially trusted assemblies.

If the following comparison is done:

 
 DECLARE @x1 xml(Test)
 DECLARE @x2 xml
 
 set @x1 = '<item xmlns="urn:test" x="00" y="2"/>'
 set @x2 = '<item xmlns="urn:test" x="00" y="2"/>'
 if dbo.CompareXml(@x1,@x2) = 1
 PRINT 'equal'
 ELSE
 PRINT 'not equal'

It will return 'not equal' because SQL Server does not store literal xml for a strongly typed xml variable. It stores strongly typed xml datatype in their XML Schema canonical form and this is '0' for '00' if the datatype is xs:int. Again this is expected because of the way that SQL Server treats strongly typed xml.

Dan

CLR Based Histogram Functions and SQL Server 2005

Dan Sullivan — Sat, 26 Aug 2006 11:43:00 GMT

CLR Based Histogram Functions and SQL Server 2005

One of the enhancements to SQL Server 2005 is the ability to create your own aggregates, or User Defined Aggregate as they are called. They are pretty easy to make and I'll go over the basics later in this article. But the real question is why would you want to make your own UDA? To answer that question we have to look some of the details of aggregates in general.

The code for the examples in this article can be found at code samples.

For example SQL Server has only a few built-in aggregates compared to Oracle and you might like to add some of the ones found in Oracle. Oracle has an aggregate named COVAR_POP that you might like to be able to use in SQL Server. However most aggregates that are basically some kind of accumulation of arithmetic calculations can be composed out of the aggregates that already exist in SQL Server. For example:

SELECT (SUM(col1 * col2) - SUM(col2) * SUM(col1) / count(*)) / count(*) FROM myTable
WHERE col1 IS NOT NULL and col2 IS NOT NULL

produces the same result as the Oracle aggregate. So for most arithmetic aggregates you need not create a CLR UDA. In general you can use a composition of the built-in aggregates in SQL Server to create most aggregates that accumlate arithmetic calculations.

A couple of kinds of aggregates that you can't compose out of the aggregate functions built into SQL Server are a median and string concatenation aggragates. I've talked about median calculations last year. You can find the blog articles at Psuedo Median Aggregrate and Yet Another Median Calculation.

The string concatenation aggregate is at least a candidate for a UDA, but a UDA has a limitation that requires its intermediate state to be serializable in 8000 bytes. You can make a UDA that concatenates strings but it seems like it would be a bit scary to use in practice. It could be working fine in production for months and then suddenly fail because it happens produce a string that is too long. I would call this brittle and avoid it, but YMMV. Also, if you don't need a aggregate function, but just an aggregate calculation, there are other ways to do this in SQL Server.

There is another kind of "aggregate" that is useful, a histogram aggregate. I put the term aggregate in quotes because all of SQL Server's built-in aggregate functions and UDA's return scalars and a histogram is a table. But we are going to look at a way to use a UDA to create a histogram and, with a little slight of hand, make it return a table. We will call this a UDA/Histogram.

There are at least two kinds of UDA/histograms we could look at. One is a histogram of the letter usage in words. For example the letter usage for the the pharse "Attention all aardvarks." would show that the letter A was used five times, assuming we ignore case. This UDA/histogram could be used against a table of customer names to find out the of letter usage all customer names.

Another historgram we could look at is more of a classic histogram, it would find the usage of numbers in a set of ranges, maybe as the count of numbers in the range 0 though 1, 1 through 2, and so on up to 9 through 10.

There are a number of ways to implement a histogram in SQL Server 2005. One is via the UDA/histogram that will will be looking at shortly, one is by JOINing with a range table, and the third is to use a CURSOR. In this article we will look at the letter histogram. In later blog articles we will look at using CLR UDA to implement more "classic" histograms.

We will look at the letter usage UDA/histogram now. First of all a UDA has four methods that you must implement: Init, Accumulate, Merge and Terminate. If you use Visual Studio and make a language/database project it will skeleton UDA for you, you will just have to fill in these methods.

This implementation has 26 Int32 fields, one for each letter in the English alphabet, to accumulate the letter counts. It might seem better to use an array to do this but if you do it is a bit more complicated to implement. A fragement of the accumlators and Init method are shown below:

public struct LetterHistogramAgg
{
 Int32 As;
 Int32 Bs;
 Int32 Cs;
...
 public void Init()
 {
  As = 0;
  Bs = 0;
...

The Accumulate method converts the incoming string to upper case, the checks out each of its letters to increment the approprate accumlator. Here is a fragement of that:

 public void Accumulate(SqlString Value)
 {
  Char[] letters = Value.Value.ToUpper().ToCharArray();
  foreach (Char letter in letters)
  {
   if (letter == 'A')
   {
    As = As + 1;
    continue;
   }

The Merge method just has to add together the accumlators, you can see that in the example code.

The terminate method has to return the histogram. Because this is a UDA that must be a scalar, so what it returns is a ";" separated string with the counts from each letter accumulator, in order. Here is a fragment of that method:

 public SqlString Terminate()
 {
  StringBuilder sb = new StringBuilder();
  sb.Append(As.ToString());
  sb.Append(";");
  sb.Append(Bs.ToString());
  sb.Append(";");

At first glance it might seem less than useful to return a histogram as a string, but shortly we will see it is very easy to make a table valued user defined function that converts it to a table.

One of the pieces of test data I have are tables of male and female first names and last names that were used to fill out United States census forms. You can get the raw data for these tables from http://www.census.gov/genealogy/names/dist.all.last, http://www.census.gov/genealogy/names/dist.female.first, and http://www.census.gov/genealogy/names/dist.male.last.

I've named the tables MaleFirstName, FemaleFirstNames, and LastNames. Using the LetterHistogramAgg on the MaleFirstName table produces:

SELECT dbo.LetterHistogramAgg(Name) from MaleFirstNames)
-------------------------------------------------------------------------
664;135;234;333;785;86;117;191;448;104;94;507;229;597;561;52;13;677;286;304;168;73;83;10;205;18;

Next lets look at making a CLR table valued user defined function that can turn this into something more useful. In order to create a table valued function using the CLR, the function must return an object with an IEnumerable interface. This can be a bit tedious to implement, but in C# the yield keyword will in fact do most of the implementation for you. Below is the code for a UDF that converts the string produced by the LetterHistorgramAgg into a table. The name of the function it implements is LetterHistogramTable.

public partial class UserDefinedFunctions
{
 public static void GetHistRow(Object obj, out Char c, out Int32 count)
 {
  HistSample hs = obj as HistSample;
  c = hs.letter;
  count = hs.count;
 }
 class HistSample
 {
  internal Char letter;
  internal Int32 count;
  internal HistSample(Char letter, Int32 count)
  {
   this.count = count;
   this.letter = letter;
  }
 }
 [Microsoft.SqlServer.Server.SqlFunction(
  FillRowMethodName="GetHistRow",
  IsPrecise=true,
  IsDeterministic=true,
  TableDefinition="Letter NCHAR(1), [Count] INT")]
 public static IEnumerable LetterHistogramTable(SqlString histogram)
 {
  if (!histogram.IsNull)
  {
   String[] counts = histogram.Value.Split(new char[] { ';' });
   String alphabet = "ABCDEFGHIJKLMNOPQUSTUVWXYZ";
   for (Int32 index = 0; index < 26; index++)
   {
    yield return new HistSample(alphabet[index], Int32.Parse(counts[index]));
   }
  }
 }
};

Once we have deployed the LetterHistogramTable UDF we can retry making the histograms of the letters in the MaleFirstNames table. We use the LetterHistogramAgg to make the histogram string, then use the LetterHistogramTable to turn the string into a table.

SELECT * FROM dbo.LetterHistogramTable((
 SELECT dbo.LetterHistogramAgg(Name) from MaleFirstNames))

Letter Count
------ -----------
A      664
B      135
C      234
D      333
E      785
F      86
G      117
H      191
I      448
J      104
K      94
L      507
M      229
N      597
O      561
P      52
Q      13
U      677
S      286
T      304
U      168
V      73
W      83
X      10
Y      205
Z      18

Let's do the same thing with the FemaleFirstNames table:

SELECT * FROM dbo.LetterHistogramTable((
 SELECT dbo.LetterHistogramAgg(Name) from FemaleFirstNames))
Letter Count
------ -----------
A      4146
B      327
C      717
D      811
E      3370
F      142
G      346
H      796
I      2350
J      311
K      405
L      2064
M      775
N      2330
O      986
P      147
Q      43
U      1793
S      1097
T      1217
U      423
V      262
W      100
X      29
Y      686
Z      94

There is another way to implement the letter historgram and that is to use a CURSOR. You create a table to hold the accumulations of letter counts and then use the CURSOR to fill that table. Here is an example:

CREATE TABLE #letterCounts
(
letter CHAR(1) PRIMARY KEY,
count INT
)
INSERT INTO #letterCounts SELECT 'A', 0
INSERT INTO #letterCounts SELECT 'B', 0
.....

UPDATE #letterCounts SET Count=0
DECLARE  Name_Cursor CURSOR FOR SELECT Name FROM MaleFirstNames
DECLARE @name VARCHAR(MAX)
OPEN Name_Cursor;
FETCH NEXT FROM Name_Cursor INTO @name;
WHILE @@FETCH_STATUS = 0
BEGIN
 DECLARE @index INT
 SET @index = LEN(@name)
 WHILE @index > 0
 BEGIN
  -- go through each letter in string
  UPDATE #letterCounts SET count = count + 1 WHERE SUBSTRING(@name, @index, 1) = letter
  SET @index = @index - 1
 END
    FETCH NEXT FROM Name_Cursor INTO @name
END;
CLOSE Name_Cursor;
DEALLOCATE name_Cursor;
SELECT * FROM #letterCounts

letter count
------ -----------
A      664
B      135
C      234
D      333
E      785
F      86
G      117
H      191
I      448
J      104
K      94
L      507
M      229
N      597
O      561
P      52
Q      13
R      677
S      286
T      304
U      168
V      73
W      83
X      10
Y      205
Z      18

In general any aggregate can be implemented using a CURSOR. The difference between using a CURSOR to implement the aggregate and a CLR user defined aggregate is speed. In general a CLR user defined aggregate will be more than two orders of magnitude, that is 100 times, faster than the same aggregate implemented using a CURSOR. On my little portable running SQL Server Developer Edition the Cursor based implementation takes about 50 seconds to run and the user defined aggregate one runs in well under one second. Of course this is just a generalization to show that CLR UDA's in general run faster than CURSOR based one... always check you data in your usage patterns before making any kind of decision about performance

This isn't the whole store on user defined aggregates but it is a start. In later blog articles we will look at "classic" aggregates and also look at some more trade-off between the implementations of aggregates.

Dan

What cost xml, SQL Server?

Dan Sullivan — Sun, 06 Aug 2006 15:36:00 GMT

How much space does xml cost in SQL Server 2005? This question seems to come up a lot because xml has a lot of meta data in it and to top it off SQL Server saves the xml data type using UTF-16 encoding. This is, at least for xml that is mostly ascii type characters, up twice as much space as the encoding you typically see for xml, UTF-8. So if you just want to save xml in SQL Server then return it later you're better off storing as VARCHAR(MAX) or maybe VARBINARY(MAX), right? Maybe not, but that is what this article is going to look at...

Even before we look at the space issues involved I'd like to mention that I think it is really being "too clever by half" to store xml as anything other than xml. Imagine if someone gave you a CSV file that looked like:

pipe, 100
rope, 250

and and told you the first column was the name of an item and the second was the price in whole dollars, then asked you to make a table and import it into SQL Server. You could make a table that looked like:

CREATE TABLE Stuff
(
name VARCHAR(MAX),
price VARCHAR(MAX)
)

Later when people when to use the table they could issue a query like this:

SELECT name, CAST(price AS INT) FROM Stuff

Of couse if one of the files you imported had a line like:

hose, XVII

the query would get an error at runtime when it ran. The problem here was that the data was stored as something that it wasn't. Storing xml as a string or binary is just like store an INT as text. When you insert something into an xml column SQL Server makes sure that is really is xml or rejects it, so you will not have any errors if you happen to make use of some of the xml functionality SQL Server 2005 provides. Even if you don't do make any use of the xml processing functionality in SQL Server, the client that reads the xml as text you've stored will get the error. It's usually best to catch errors as early as possible, especially if the client thinks your server made the error :-).

OK, now lets look at how much space things really take up. To do this we are going to make four tables that store xml each in a different way, fill them up with a bunch of xml then see how big they are. You can down load the code for trying these experiments yourself from http://www.pluralsight.com/dan/samples/whatcostxml.zip.

-- save as binary
CREATE TABLE XmlBinSizeTest
(
data VARBINARY(MAX)
)

--save as varchar
CREATE TABLE XmlTextSizeTest
(
data VARCHAR(MAX)
)

--save save as xml with a schema
CREATE TABLE XmlSchemaSizeTest
(
data xml(XmlSizeSchema)
)

-- save as xml without a schema
CREATE TABLE XmlSizeTest
(
data xml
)

Now we want to fill these tables up with a big walop of xml. When I need some static string or xml for testing I usually wrap it in a function so I don't have to put the literal text in my SQL expressions.

CREATE
--ALTER
FUNCTION XmlSizeTestData4()
RETURNS VARCHAR(MAX)
AS
BEGIN
RETURN
'http://www.w3.org/2001/XMLSchema-instance">

guy1@adventure-works.com" EmailPromotion="0" Phone="320-555-0195" PasswordHash="1196EB7D5425B281CACDCDD2F60F52D9689D9E49" PasswordSalt="Lanmhoo=" rowguid="D366A33A-8EDE-42BD-BF79-3E7FB9713FE1" ModifiedDate="1996-07-24T00:00:00"/>

'
END

Then a little batch to fill the tables:

DECLARE @t VARCHAR(MAX)
SET @t = dbo.XmlSizeTestData4();
DECLARE @index INT
SET @index = 100
WHILE @index > 0
BEGIN
SET @index = @index - 1
INSERT INTO XmlTextSizeTest VALUES (@t);
INSERT INTO XmlSizeTest VALUES (@t);
INSERT INTO XmlSchemaSizeTest VALUES (@t);
INSERT INTO XmlBinSizeTest VALUES (CAST(@t AS VARBINARY(MAX)));
END

We can use sys.allocation_units to find out how much space the tables take up. It takes a few joins to make this work off of table names...

SELECT o.name AS table_name, au.type_desc, au.used_pages
FROM sys.allocation_units AS au
JOIN sys.partitions AS p ON au.container_id = p.partition_id
JOIN sys.objects AS o ON p.object_id = o.object_id
WHERE o.name in (N'XmlTextSizeTest',
N'XmlSizeTest',
N'XmlSchemaSizeTest',
N'XmlBinSizeTest')

In SQL Server 2005 large objects, i.e. things like xml and VARCHAR(MAX) overflow in to supplementary pages outside of the pages for the table itself. This query tells us about both the pages used in the table and the extra overflow pages. Note that DBCC CHECKTABLE won't give you the right answer, because it won't tell you about the overflow pages. Running this query produces:

table name               type_description     used pages
XmlBinSizeTest         IN_ROW_DATA     2
XmlBinSizeTest         LOB_DATA             2802
XmlTextSizeTest       IN_ROW_DATA      2
XmlTextSizeTest       LOB_DATA             2802
XmlSchemaSizeTest IN_ROW_DATA     2
XmlSchemaSizeTest LOB_DATA            4226
XmlSizeTest              IN_ROW_DATA     2
XmlSizeTest              LOB_DATA           2907

The numbers tell an interesting story. First of all no suprise, storing text as binary takes the same amount of space as storing it as VARCHAR(MAX). However storing xml without a schema takes only about 5% more space than storing it as VARCHAR(MAX) even though the xml is being stored as UTF-16, not single byte characters as VARCHAR does. How can this be?

Let's use different xml data that really exacerbates the difference between storing xml as text and storing xml as xml. Here is another function that cobbles up some xml on the fly:

CREATE
FUNCTION XmlTestData1()
RETURNS VARCHAR(MAX)
AS
BEGIN
DECLARE @t VARCHAR(MAX)
SET @t = CAST('+ REPLICATE(CAST(' ' AS VARCHAR(MAX)), 10000)
+ '>abc'
RETURN @t
END

It's basically inserting a lot of ignorable space into the the xml. A fragment of the xml it generates looks like:

Actually it has a lot more spaces than that. To try this next you should drop all the tables and recreate them, then use the batch that fill the tables, but change the source of the xml to dbo.XmlTestData1(). After you run that batch that finds the sizes you see something like:

table name               type_description     used pages
XmlBinSizeTest         IN_ROW_DATA      2
XmlBinSizeTest         LOB_DATA            252
XmlTextSizeTest       IN_ROW_DATA      2
XmlTextSizeTest       LOB_DATA            252
XmlSchemaSizeTest IN_ROW_DATA     3
XmlSchemaSizeTest LOB_DATA           0
XmlSizeTest              IN_ROW_DATA 3
XmlSizeTest              LOB_DATA           0

It looks like saving this xml as text or binary takes up about two orders of magnitude more space than just saving it as xml!

When SQL Server saves xml it need not save it as literal text, and it doen't. It doesn't have to save the the "pointies" and it doesn't have to save whitespace that doesn't matter. And most of the the time the whitespace between consecutive opening tags doesn't matter. In other words SQL Server 2005 goes out of its way to efficiently store xml.

So it not a slam dunk to figure out which way of storing xml takes up the least amount of space. The point here isn't that xml will always take about the same or less as a corresponding VARCHAR(MAX), because sometimes the VARCHAR(MAX) or VARBINARY(MAX) will take up less space. But you will probably find that the difference isn't as great as you might suspect. In any case this is a database we are talking about, you should store xml as xml.

Dan