I’ve been building a number of scripts that query information using PowerShell.

To create those, I used some PowerShell tricks to make the output look just like I wanted. This post is a compilation of these tricks.

The first few tricks are simple, but they grow increasingly complex towards the end.

The examples here are geared towards SMB Shares and Volumes, but you can apply these concepts in many, many other situations.

1) Using –AutoSize

I do not like the default way that PowerShell spaces out fields in a query. I prefer them packed on the left, which is what the AutoSize option of the Format-Table cmdlet (FT for short) does.

AutoSize, as the name implies, also adjusts to the size of the properties and avoids showing “…” at the end of the output.

PS C:\> Get-SmbShare

Name                          ScopeName                     Path                          Description
----                          ---------                     ----                          -----------
ADMIN$                        *                             C:\windows                    Remote Admin
BackupFiles                   *                             E:\Backup                     Project backup files
C$                            *                             C:\                           Default share
D$                            *                             D:\                           Default share
F$                            *                             F:\                           Default share
Images                        *                             D:\Images                     Pictures to be used in pro...
IPC$                          *                                                           Remote IPC
print$                        *                             C:\windows\system32\spool\... Printer Drivers
Projects                      *                             C:\Projects                   Recent project files

PS C:\> Get-SmbShare | FT -AutoSize

Name        ScopeName Path                              Description
----        --------- ----                              -----------
ADMIN$      *         C:\windows                        Remote Admin
BackupFiles *         E:\Backup                         Project backup files
C$          *         C:\                               Default share
D$          *         D:\                               Default share
F$          *         F:\                               Default share
Images      *         D:\Images                         Pictures to be used in projects
IPC$        *                                           Remote IPC
print$      *         C:\windows\system32\spool\drivers Printer Drivers
Projects    *         C:\Projects                       Recent project files

2) Selecting columns to show

Every object has a default view with a specific set of columns. If you don’t like those, you can select your own. To find out what fields you can use, you can use a “Select *” or a “Get-Member” to find out.

PS C:\> Get-SmbShare | Select * -First 1

PresetPathAcl         :
ShareState            : Online
AvailabilityType      : NonClustered
ShareType             : FileSystemDirectory
FolderEnumerationMode : Unrestricted
CachingMode           : Manual
SmbInstance           : Default
CATimeout             : 0
ConcurrentUserLimit   : 0
ContinuouslyAvailable : False
CurrentUsers          : 0
Description           : Remote Admin
EncryptData           : False
Name                  : ADMIN$
Path                  : C:\windows
Scoped                : False
ScopeName             : *
SecurityDescriptor    : O:SYG:SYD:(A;;GA;;;BA)(A;;GA;;;BO)(A;;GA;;;IU)
ShadowCopy            : False
Special               : True
Temporary             : False
Volume                : \\?\Volume{4304337d-6763-11e3-8255-806e6f6e6963}\
PSComputerName        :
CimClass              : ROOT/Microsoft/Windows/SMB:MSFT_SmbShare
CimInstanceProperties : {AvailabilityType, CachingMode, CATimeout, ConcurrentUserLimit...}
CimSystemProperties   : Microsoft.Management.Infrastructure.CimSystemProperties

PS C:\> Get-SmbShare | Get-Member

   TypeName: Microsoft.Management.Infrastructure.CimInstance#ROOT/Microsoft/Windows/SMB/MSFT_SmbShare

Name                      MemberType     Definition
----                      ----------     ----------
Clone                     Method         System.Object ICloneable.Clone()
Dispose                   Method         void Dispose(), void IDisposable.Dispose()
Equals                    Method         bool Equals(System.Object obj)
GetCimSessionComputerName Method         string GetCimSessionComputerName()
GetCimSessionInstanceId   Method         guid GetCimSessionInstanceId()
GetHashCode               Method         int GetHashCode()
GetObjectData             Method         void GetObjectData(System.Runtime.Serialization.SerializationInfo info, Sys...
GetType                   Method         type GetType()
ToString                  Method         string ToString()
CATimeout                 Property       uint32 CATimeout {get;set;}
ConcurrentUserLimit       Property       uint32 ConcurrentUserLimit {get;set;}
ContinuouslyAvailable     Property       bool ContinuouslyAvailable {get;set;}
CurrentUsers              Property       uint32 CurrentUsers {get;set;}
Description               Property       string Description {get;set;}
EncryptData               Property       bool EncryptData {get;set;}
Name                      Property       string Name {get;}
Path                      Property       string Path {get;}
PSComputerName            Property       string PSComputerName {get;}
Scoped                    Property       bool Scoped {get;}
ScopeName                 Property       string ScopeName {get;}
SecurityDescriptor        Property       string SecurityDescriptor {get;set;}
ShadowCopy                Property       bool ShadowCopy {get;}
Special                   Property       bool Special {get;}
Temporary                 Property       bool Temporary {get;}
Volume                    Property       string Volume {get;}
AvailabilityType          ScriptProperty System.Object AvailabilityType {get=[Microsoft.PowerShell.Cmdletization.Gen...
CachingMode               ScriptProperty System.Object CachingMode {get=[Microsoft.PowerShell.Cmdletization.Generate...
FolderEnumerationMode     ScriptProperty System.Object FolderEnumerationMode {get=[Microsoft.PowerShell.Cmdletizatio...
PresetPathAcl             ScriptProperty System.Object PresetPathAcl {get=$acl = Get-Acl ($this.PSBase.CimInstancePr...
ShareState                ScriptProperty System.Object ShareState {get=[Microsoft.PowerShell.Cmdletization.Generated...
ShareType                 ScriptProperty System.Object ShareType {get=[Microsoft.PowerShell.Cmdletization.GeneratedT...
SmbInstance               ScriptProperty System.Object SmbInstance {get=[Microsoft.PowerShell.Cmdletization.Generate...

PS C:\> Get-SmbShare | FT Name, Path, Special, AvailabilityType -AutoSize

Name        Path                              Special AvailabilityType
----        ----                              ------- ----------------
ADMIN$      C:\windows                           True     NonClustered
BackupFiles E:\Backup                           False     NonClustered
C$          C:\                                  True     NonClustered
D$          D:\                                  True     NonClustered
F$          F:\                                  True     NonClustered
Images      D:\Images                           False     NonClustered
IPC$                                             True     NonClustered
print$      C:\windows\system32\spool\drivers   False     NonClustered
Projects    C:\Projects                         False     NonClustered

3) Selecting rows to show

You can restrict the items to show based on any of its properties. You need to learn to use the many types of operators like equal (-eq), not equal (-ne), greater than (-gt), like (-like), not like (-not like) and many others.

The cmdlet used is Where-Object, but it can be abbreviated as Where or simply a question mark. There is a simple form for querying just one property and also a more complex form for using expressions.

For instance, to show all shares that are not special, you can one of these:
Get-SmbShare | Where-Object {$_.Special –ne $true}
Get-SmbShare | ? Special –ne $true

If you use the form with the expression in {}, you must specify $_.property, where $_ means “the current item we’re processing”. The simpler form without {} can be used with only one property.

Here’s a more complex example to show all shares that are not special and with a description starting with the letter P:
Get-SmbShare | ? { $_.Special -ne $true -and $_.Description -like "P*" }

PS C:\> Get-SmbShare | Where-Object {$_.Special –ne $true} | FT -AutoSize

Name        ScopeName Path                              Description
----        --------- ----                              -----------
BackupFiles *         E:\Backup                         Project backup files
Images      *         D:\Images                         Pictures to be used in projects
print$      *         C:\windows\system32\spool\drivers Printer Drivers
Projects    *         C:\Projects                       Recent project files

PS C:\> Get-SmbShare | ? Special –ne $true| FT -AutoSize

Name        ScopeName Path                              Description
----        --------- ----                              -----------
BackupFiles *         E:\Backup                         Project backup files
Images      *         D:\Images                         Pictures to be used in projects
print$      *         C:\windows\system32\spool\drivers Printer Drivers
Projects    *         C:\Projects                       Recent project files

PS C:\> Get-SmbShare | ? Special | FT -AutoSize

Name   ScopeName Path       Description
----   --------- ----       -----------
ADMIN$ *         C:\windows Remote Admin
C$     *         C:\        Default share
D$     *         D:\        Default share
F$     *         F:\        Default share
IPC$   *                    Remote IPC

PS C:\> Get-SmbShare | ? { $_.Special -ne $true -and $_.Description -like "P*" } | FT -AutoSize

Name        ScopeName Path                              Description
----        --------- ----                              -----------
BackupFiles *         E:\Backup                         Project backup files
Images      *         D:\Images                         Pictures to be used in projects
print$      *         C:\windows\system32\spool\drivers Printer Drivers

4) Creating custom columns

In addition to the ability to select which columns to show, you can also create custom tables with any expression. This is useful for creating new calculated columns based on the existing ones.

You can also use this same process to rename existing properties (if you don’t like their default names) or customize the alignment/size of the column.

As with the full form of the “Where” filters, the expressions here must be enclosed in {} and references to existing properties must be preceded with $_.

The syntax uses a hash table, which is little unusual. You should include at least a “Label” (or “Name”) and an “Expression” item in the hash table. You can also specify “Alignment” (or “Align”) and “Width”.

Here’s an example to rename the “Path” property to “Folder”:
Get-SmbShare | FT Name, @{ Label=”Folder”; Expression={$_.Path} }, Description –AutoSize

Here’s another example showing only the drive letter of the path:
Get-SmbShare | FT Name, Description, @{ Name=”Drive”; Expression={$_.Path.Substring(0,1)}; Alignment=”Center” } –AutoSize

Lastly, a more complex example showing that shares ending with $ are hidden:
Get-SmbShare | FT Name, Path, @{ Align="Center"; Expression={If ($_.Name –like “*$”) {“Yes”} else {“No”} }; Label=”Hidden” } –AutoSize

It’s a lot of curly braces, I know. Just make sure you keep good track of them.

PS C:\> Get-SmbShare | FT Name, @{ Label=”Folder”; Expression={$_.Path} }, Description –AutoSize

Name        Folder                            Description
----        ------                            -----------
ADMIN$      C:\windows                        Remote Admin
BackupFiles E:\Backup                         Project backup files
C$          C:\                               Default share
D$          D:\                               Default share
F$          F:\                               Default share
Images      D:\Images                         Pictures to be used in projects
IPC$                                          Remote IPC
print$      C:\windows\system32\spool\drivers Printer Drivers
Projects    C:\Projects                       Recent project files

 
PS C:\> Get-SmbShare | FT Name, Description, @{ Name=”Drive”; Expression={$_.Path.Substring(0,1)}; Alignment=”Center” } –AutoSize

Name        Description                     Drive
----        -----------                     -----
ADMIN$      Remote Admin                      C
BackupFiles Project backup files              E
C$          Default share                     C
D$          Default share                     D
F$          Default share                     F
Images      Pictures to be used in projects   D
IPC$        Remote IPC
print$      Printer Drivers                   C
Projects    Recent project files              C

PS C:\> Get-SmbShare | FT Name, Path, @{ Align="Center"; Expression={If ($_.Name –like “*$”) {“Yes”} else {“No”} }; Label=”Hidden” } –AutoSize

Name        Path                              Hidden
----        ----                              ------
ADMIN$      C:\windows                         Yes
BackupFiles E:\Backup                           No
C$          C:\                                Yes
D$          D:\                                Yes
F$          F:\                                Yes
Images      D:\Images                           No
IPC$                                           Yes
print$      C:\windows\system32\spool\drivers  Yes
Projects    C:\Projects                         No

PS C:\> Get-Volume | Sort DriveLetter | FT DriveType, DriveLetter, FileSystem, @{Expression={ [int] ($_.Size/1MB) }; Label="Total (MB)"; Align="Right" }, @{Expression={ [int] (($_.Size-$_.SizeRemaining)/1MB) }; Label="Used (MB)"; Align="Right" }, @{Expression={ [int] ($_.SizeRemaining/1MB) }; Label="Free (MB)"; Align="Right" }, @{Expression={ [int]($_.SizeRemaining/$_.Size*100) }; Label="Free %"; Align="Right" } –AutoSize

DriveType DriveLetter FileSystem Total (MB) Used (MB) Free (MB) Free %
--------- ----------- ---------- ---------- --------- --------- ------
Fixed                 NTFS              350       287        63     18
Fixed               C NTFS           243845    214308     29537     12
Fixed               D NTFS           476937     56928    420009     88
Removable           E NTFS           125903     20925    104978     83
Fixed               F NTFS           476938    137181    339757     71

PS C:\> dir d:\*.* -Recurse | Select @{Expression={$_.CreationTime.Year};Label="YearCreated"}, @{Expression={$_.CreationTime.Month};Label="MonthCreated"} | Group YearCreated, MonthCreated | Select @{Expression={$_.Name.Split(",")[0].Trim()};Label="Year"}, @{Expression={$_.Name.Split(",")[1].Trim()};Label="Month"}, Count | Sort Count -Descending | FT Year, Month, Count -AutoSize

Year Month Count
---- ----- -----
2013 7     10172
2013 8      9114
2013 10     9097
2013 9      4075
2014 2       483
2013 4        47
2012 2        16
2013 11       15
2013 6         5
2013 12        4
2014 4         3
2013 5         2
2012 12        2
2012 5         1

5) Formatting numeric fields

One interesting trick when creating expressions is to format numbers. There is a specific –f operator that you can use to format numbers with comma separators, fixed number of decimal places, currency format and also percentages.

The format string can include references to multiple numbers, so you must specify the index for the number in {}. Here’s an example using 3 numbers.

PS C:\> "Note that {1} times {0} equals {2}" -f 123, 2, 246
Note that 2 times 123 equals 246

Beside the index, you can can specify a colon plus a letter indicating the type of formatting followed by the number of decimal points to use. Types of formatting include C for currency, N for Numeric (just comma separators), P for Percentage and E for Exponential. Here are a few examples:

PS C:\> "They spent {0:C2} of the total {1:C2} they had. They spent {2:P2}." -f 12.34, 45.67, (12.34/45.67)
They spent $12.34 of the total $45.67 they had. They spent 27.02 %.

PS C:\> "The number {0:N2} in exponential notation is {0:E6}" -f 123456.789
The number 123,456.79 in exponential notation is 1.234568E+005

Now if you combine this new formatting feature with the expressions the previous items, you can get quite powerful results:

PS C:\> Get-Volume | Sort DriveLetter | FT DriveType, DriveLetter, FileSystem, @{Expression={ "{0:N0}" -f ($_.Size/1MB) }; Label="Total (MB)"; Align="Right" }, @{Expression={ "{0:N0}" -f (($_.Size-$_.SizeRemaining)/1MB) }; Label="Used (MB)"; Align="Right" }, @{Expression={ "{0:N0}" -f ($_.SizeRemaining/1MB) }; Label="Free (MB)"; Align="Right" }, @{Expression={ "{0:P2}" -f ($_.SizeRemaining/$_.Size) }; Label="Free %"; Align="Right" } -AutoSize

DriveType DriveLetter FileSystem Total (MB) Used (MB) Free (MB)  Free %
--------- ----------- ---------- ---------- --------- ---------  ------
Fixed                 NTFS              350       287        63 18.09 %
Fixed               C NTFS          243,845   213,652    30,193 12.38 %
Fixed               D NTFS          476,937    56,928   420,009 88.06 %
Removable           E NTFS          125,903    20,925   104,978 83.38 %
Fixed               F NTFS          476,938   137,181   339,757 71.24 %

6) Linking two tables

One useful trick is to find the relationship between objects and create a single query that shows information coming from both of them.

For instance, you might want to show the free space in the volume when showing information about a share. Or you might want to show all volumes with their associated shares.

This usually involves looping through the items in one table using the For-Each cmdlet, which can be abbreviated as %. This also involves using $_ to refer to the properties of the table being looped through.

In other to show the correct association, you commonly need to use matching or related properties from both tables. If you come from the relational database world, think of a foreign key.

Because you’re dealing with two tables, you might need to assign the outer $_ to a specific variable, so you can compare fields from the outer table with the inner table.

Here’s an example showing all volumes and their associated file shares:
Get-Volume | Sort DriveLetter | % { $V=$_; $V | FT -AutoSize; Get-SmbShare | ? {$V.ObjectID –eq $_.Volume} | FT Name, Path -AutoSize }

Here’s another example showing all shares and the free space on the associated volume:
Get-SmbShare | ? Special -ne $true | % { $_ | FT Name, Path -AutoSize; Get-Volume –ObjectID $_.Volume | FT DriveLetterSize, SizeRemaining}

PS C:\> Get-Volume | Sort DriveLetter | % { $V=$_; $V | FT -AutoSize; Get-SmbShare | ? {$V.ObjectID –eq $_.Volume} | FT Name, Path -AutoSize }

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining   Size
----------- --------------- ---------- --------- ------------ -------------   ----
            System Reserved NTFS       Fixed     Healthy            63.3 MB 350 MB

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining      Size
----------- --------------- ---------- --------- ------------ -------------      ----
C           SSD1            NTFS       Fixed     Healthy           29.49 GB 238.13 GB

Name      Path
----      ----
ADMIN$    C:\windows
C$        C:\
print$    C:\windows\system32\spool\drivers
Projects  C:\Projects

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining      Size
----------- --------------- ---------- --------- ------------ -------------      ----
D           HDD             NTFS       Fixed     Healthy          410.17 GB 465.76 GB

Name   Path
----   ----
D$     D:\
Images D:\Images

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining      Size
----------- --------------- ---------- --------- ------------ -------------      ----
E           SDCard          NTFS       Removable Healthy          102.52 GB 122.95 GB

Name        Path
----        ----
BackupFiles E:\Backup

DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining      Size
----------- --------------- ---------- --------- ------------ -------------      ----
F           USB3            NTFS       Fixed     Healthy          331.79 GB 465.76 GB

Name Path
---- ----
F$   F:\

PS C:\> Get-SmbShare | ? Special -ne $true | % { $_ | FT Name, Path -AutoSize; Get-Volume –ObjectID $_.Volume | FT DriveLetter, Size, SizeRemaining}

Name        Path
----        ----
BackupFiles E:\Backup

DriveLetter         Size SizeRemaining
-----------         ---- -------------
          E 132018860032  110077665280

Name   Path
----   ----
Images D:\Images

DriveLetter         Size SizeRemaining
-----------         ---- -------------
          D 500104687616  440411648000

Name   Path
----   ----
print$ C:\windows\system32\spool\drivers

DriveLetter         Size SizeRemaining
-----------         ---- -------------
          C 255690010624   31659585536

Name     Path
----     ----
Projects C:\Projects

DriveLetter         Size SizeRemaining
-----------         ---- -------------
          C 255690010624   31659585536

7) Creating a few fake fields

Another useful trick is to create a temporary object with a few fake properties that you can fill in with some scripting. This requires using variables to store the temporary results and a little more understanding of programming structures like loops.

This provides a better way to combine two related tables into a single view, like we did in the previous item. But this time we get a single, combined output.

For instance, Get-SmbShare will return a volume ID that you can link to a volume, but it would be nice if we could have a single query with some properties from the share and some from the volume.

Here’s what you would do: First, create a variable with the result of Get-SmbShare, including the properties you want to fill later. These won’t exist initially, so they start empty. Then we’ll loop through the items and fill those empty properties using a query for the volume ID. After that, the results are ready to be formatted using any of our previous tricks…

PS C:\> $Shares = Get-SmbShare | ? { $_.Special -ne $true} | Select Name, Path, Volume, Total, Used, Free
 
PS C:\> $Shares | FT -AutoSize

Name        Path                              Volume                                            Total Used Free
----        ----                              ------                                            ----- ---- ----
BackupFiles E:\Backup                         \\?\Volume{a68e6379-6763-11e3-8256-d89d67d108b9}\
Images      D:\Images                         \\?\Volume{4304337f-6763-11e3-8255-806e6f6e6963}\
print$      C:\windows\system32\spool\drivers \\?\Volume{4304337d-6763-11e3-8255-806e6f6e6963}\
Projects    C:\Projects                       \\?\Volume{4304337d-6763-11e3-8255-806e6f6e6963}\

PS C:\> $Shares | % { $S=$_; $V=Get-Volume | ? ObjectId -eq $S.Volume; $S.Total=$V.Size/1MB; $S.Free=$V.SizeRemaining/1MB; $S.Used=($V.Size-$V.SizeRemaining)/1MB }
 
PS C:\> $Shares | Sort Name | FT Name, Path, @{Expression={ "{0:N0}" -f $_.Total }; Label="Total (MB)"; Align="Right" }, @{Expression={ "{0:N0}" -f $_.Used }; Label="Used (MB)"; Align="Right" }, @{Expression={ "{0:N0}" -f $_.Free }; Label="Free (MB)"; Align="Right" }, @{Expression={ "{0:P2}" -f ($_.Free/$_.Total) }; Label="Free %"; Align="Right" } -AutoSize

Name        Path                              Total (MB) Used (MB) Free (MB)  Free %
----        ----                              ---------- --------- ---------  ------
BackupFiles E:\Backup                            125,903    20,925   104,978 83.38 %
Images      D:\Images                            476,937    56,928   420,009 88.06 %
print$      C:\windows\system32\spool\drivers    243,845   213,652    30,193 12.38 %
Projects    C:\Projects                          243,845   213,652    30,193 12.38 %

8) Creating entirely fake tables

You can also create entirely new objects using a trick similar to the previous one. You start by creating a new item simply piping an empty variable to a Select cmdlet that specifies the columns you want to create.

PS C:\> $Alerts = "" | Select Severity, Entity, Instance, Description

Then you can use any cmdlets to populate that item’s columns and sent them out.

PS C:\> Get-Volume | ? {($_.SizeRemaining/$_.Size) -lt 0.8} | % { $Alerts.Severity="Warning"; $Alerts.Entity="Volume"; $Alerts.Instance = $_.DriveLetter+" "+$_.ObjectID; $Alerts.Description="Volume has less than 80% free space"; $Alerts} | FT -AutoSize

Severity Entity Instance                                            Description
-------- ------ --------                                            -----------
Warning  Volume   \\?\Volume{4304337c-6763-11e3-8255-806e6f6e6963}\ Volume has less than 80% free space
Warning  Volume F \\?\Volume{c9337eca-6774-11e3-825a-b8763fd91487}\ Volume has less than 80% free space
Warning  Volume C \\?\Volume{4304337d-6763-11e3-8255-806e6f6e6963}\ Volume has less than 80% free space

This gets even more useful if you pack it as a PowerShell function, which you can name just like a regular cmdlet:

Function Get-StorageAlert
{

    $A = "" | Select Severity, Entity, Instance, Description

    Get-Volume | ? DriveLetter | ? {($_.SizeRemaining/$_.Size) -lt 0.8} | % {
        $A.Severity="Warning"
        $A.Entity="Volume"
        $A.Instance = $_.DriveLetter
        $A.Description="Volume has less than 80% free space"
        $A
    }

    Get-SmbShare -ContinuouslyAvailable $false | ? {$_.Special -ne $true} | % {
        $A.Severity="Warning"
        $A.Entity="Share"
        $A.Instance = $_.Name
        $A.Description="Share is not continuously available"
        $A
    }

}

Here’s a sample output:

PS C:\> Get-StorageAlert | FT -AutoSize

Severity Entity             Instance Description
-------- ------             -------- -----------
Warning  Volume                    F Volume has less than 80% free space
Warning  Volume                    C Volume has less than 80% free space
Warning  Share           BackupFiles Share is not continuously available
Warning  Share                Images Share is not continuously available
Warning  Share                print$ Share is not continuously available
Warning  Share              Projects Share is not continuously available

During my TechEd 2014 presentation on File Server Networking for a Private Cloud Storage Infrastructure in Windows Server 2012 R2, I did a demo showing a Scale-Out File Server Cluster. The demo consisted of showing the networking details of this cluster, looking both from the cluster perspective and the client side. I was asked after the presentation to share the details of the PowerShell script that I used. So, here it goes.

First, here is the output of the script. It’s broken into a few parts:

• It starts by showing the cluster configuration (cluster nodes, cluster networks, cluster resources and cluster shared volumes)
• Note that the name of the cluster itself is JOSE-S and the name of the Scale-Out File Server is JOSE-SO.
• Then it shows the IP addresses on each node, plus what’s registered on DNS for the scale-out file server name.
• Next, it outputs the SMB server network information, with focus on the scopes defined on the server.
• Then we switch to the client view of connections, after just showing the directory from the 3 shares created. This shows that multichannel won’t engage until you have a read or a write.
• Finally, after running a quick workload on the shares, it shows the full set of 24 connections (3 shares * 2 NICs * 4 connections per NIC).
• The 10.1.x.100 IP addresses are associated with the cluster name used for cluster management. They don't relate directly to the Scale-Out File Server.
• The scope name starting with "FE80" is the internal cluster scope used for inter-node communication.

Cluster Information

PS C:\> Get-ClusterNode -Cluster JOSE-S

Name    ID State
----    -- -----
JOSE-S1 2  Up
JOSE-S2 1  Up
JOSE-S3 3  Up

PS C:\> Get-ClusterNetwork -Cluster JOSE-S

Cluster Name              State Address  Role
------- ----              ----- -------  ----
JOSE-S  Cluster Network 1    Up 10.1.2.0    3
JOSE-S  Cluster Network 2    Up 10.1.1.0    3

PS C:\> Get-ClusterResource -Cluster JOSE-S

Name                              State  OwnerGroup    ResourceType
----                              -----  ----------    ------------
Cluster Disk 1                    Online Cluster Group Physical Disk
Cluster IP Address                Online Cluster Group IP Address
Cluster IP Address 10.1.1.100     Online Cluster Group IP Address
Cluster Name                      Online Cluster Group Network Name
JOSE-SO                           Online JOSE-SO       Distributed Network Name
Scale-Out File Server (\\JOSE-SO) Online JOSE-SO       Scale Out File Server

PS C:\> Get-ClusterSharedVolume -Cluster JOSE-S

Name           State  Node
----           -----  ----
Cluster Disk 2 Online JOSE-S2
Cluster Disk 3 Online JOSE-S1
Cluster Disk 4 Online JOSE-S3

IP Addresses for each node, DNS for Scale-Out Name

PS C:\> Get-NetIPAddress -CimSession <node>

PSComputerName ifIndex IPAddress  PrefixLength
-------------- ------- ---------  ------------
JOSE-S1             12 10.1.1.100           24
JOSE-S1             12 10.1.1.2             24
JOSE-S1             14 10.1.2.100           24
JOSE-S1             14 10.1.2.2             24

PSComputerName ifIndex IPAddress PrefixLength
-------------- ------- --------- ------------
JOSE-S2             12 10.1.1.3            24
JOSE-S2             14 10.1.2.3            24

PSComputerName ifIndex IPAddress PrefixLength
-------------- ------- --------- ------------
JOSE-S3             12 10.1.1.4            24
JOSE-S3             14 10.1.2.4            24

PS C:\> Resolve-DNSName JOSE-SO

Name              Type TTL  Section IPAddress
----              ---- ---  ------- ---------
JOSE-SO.JOSE.TEST A    1200 Answer  10.1.1.3
JOSE-SO.JOSE.TEST A    1200 Answer  10.1.1.2
JOSE-SO.JOSE.TEST A    1200 Answer  10.1.1.4
JOSE-SO.JOSE.TEST A    1200 Answer  10.1.2.3
JOSE-SO.JOSE.TEST A    1200 Answer  10.1.2.4
JOSE-SO.JOSE.TEST A    1200 Answer  10.1.2.2

IP Addresses on each node per Scope name

PS C:\> Get-SmbServerNetworkInterface -CimSession JOSE-SO

PSComputerName ScopeName                 IPAddress
-------------- ---------                 ---------
JOSE-S1        FE80::9CBF:78F5:DFA8:803B 10.1.2.2
JOSE-S1        FE80::9CBF:78F5:DFA8:803B 10.1.1.2
JOSE-S1        JOSE-S                    10.1.2.100
JOSE-S1        JOSE-S                    10.1.1.100
JOSE-S1        JOSE-SO                   10.1.2.2
JOSE-S1        JOSE-SO                   10.1.1.2

PSComputerName ScopeName                 IPAddress
-------------- ---------                 ---------
JOSE-S2        FE80::69AF:5813:D729:CEFB 10.1.1.3
JOSE-S2        FE80::69AF:5813:D729:CEFB 10.1.2.3
JOSE-S2        JOSE-SO                   10.1.2.3
JOSE-S2        JOSE-SO                   10.1.1.3

PSComputerName ScopeName                 IPAddress
-------------- ---------                 ---------
JOSE-S3        FE80::F4BE:F77B:7EAC:6CED 10.1.1.4
JOSE-S3        FE80::F4BE:F77B:7EAC:6CED 10.1.2.4
JOSE-S3        JOSE-SO                   10.1.2.4
JOSE-S3        JOSE-SO                   10.1.1.4

Simple connection just to check directory

PS C:\> Dir \\JOSE-SO\<share>

    Directory: \\JOSE-SO\Share1

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---          5/4/2014   5:40 PM 1717986918 testfile.dat
                                           4

    Directory: \\JOSE-SO\Share2

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---          5/4/2014   5:40 PM 1717986918 testfile.dat
                                           4

    Directory: \\JOSE-SO\Share3

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---          5/4/2014   5:40 PM 1717986918 testfile.dat
                                           4

SMB Connection

PS C:\> Get-SmbConnection

ServerName ShareName Dialect Redirected Credential
---------- --------- ------- ---------- ----------
JOSE-SO    Share1    3.02          True JOSE.TEST\Administrator
JOSE-SO    Share2    3.02          True JOSE.TEST\Administrator
JOSE-SO    Share3    3.02          True JOSE.TEST\Administrator

TCP connections used by the client

PS C:\> Get-NetTCPConnection -RemotePort 445

LocalAddress RemoteAddress LocalPort RemotePort
------------ ------------- --------- ----------
10.1.1.5     10.1.1.2          49453        445
10.1.1.5     10.1.1.3          49470        445
10.1.1.5     10.1.1.4          49461        445

Putting some load on the shares

PS C:\> C:\SQLIO\SQLIO.EXE -s10 -kR -frandom -b8 -t8 -o16 -LS -BN <file>
sqlio v1.5.SG
using system counter for latency timings, 10000000 counts per second
8 threads reading for 10 secs from files \\JOSE-SO\Share1\testfile.dat, \\JOSE-SO\Share2\testfile.dat and \\JOSE-SO\Share3\testfile.dat
        using 8KB random IOs
        enabling multiple I/Os per thread with 16 outstanding
        buffering set to not use file nor disk caches (as is SQL Server)
using current size: 16384 MB for file: \\JOSE-SO\Share1\testfile.dat
using current size: 16384 MB for file: \\JOSE-SO\Share2\testfile.dat
using current size: 16384 MB for file: \\JOSE-SO\Share3\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 26822.20
MBs/sec:   209.54
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 13
Max_Latency(ms): 171
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%:  0  0  0  0  1  2  4  6  8  8  9  8  8  7  6  5  5  4  3  3  2  2  1  1  7

SMB Connection

PS C:\> Get-SmbConnection

ServerName ShareName Dialect Redirected Credential
---------- --------- ------- ---------- ----------
JOSE-SO    Share1    3.02          True JOSE.TEST\Administrator
JOSE-SO    Share2    3.02          True JOSE.TEST\Administrator
JOSE-SO    Share3    3.02          True JOSE.TEST\Administrator

SMB Multichannel Connection

PS C:\> Get-SmbMultichannelConnection

ClientIPAddress ServerIPAddress CurrentChannels
--------------- --------------- ---------------
10.1.1.5        10.1.1.2                      4
10.1.1.5        10.1.1.3                      4
10.1.1.5        10.1.1.4                      4
10.1.2.5        10.1.2.2                      4
10.1.2.5        10.1.2.3                      4
10.1.2.5        10.1.2.4                      4

TCP connections used by the client

PS C:\> Get-NetTCPConnection -RemotePort 445

LocalAddress RemoteAddress LocalPort RemotePort
------------ ------------- --------- ----------
10.1.1.5     10.1.1.2          49511        445
10.1.1.5     10.1.1.2          49513        445
10.1.1.5     10.1.1.2          49516        445
10.1.1.5     10.1.1.2          49480        445
10.1.1.5     10.1.1.3          49518        445
10.1.1.5     10.1.1.3          49497        445
10.1.1.5     10.1.1.3          49512        445
10.1.1.5     10.1.1.3          49515        445
10.1.1.5     10.1.1.4          49514        445
10.1.1.5     10.1.1.4          49517        445
10.1.1.5     10.1.1.4          49492        445
10.1.1.5     10.1.1.4          49509        445
10.1.2.5     10.1.2.2          49498        445
10.1.2.5     10.1.2.2          49501        445
10.1.2.5     10.1.2.2          49508        445
10.1.2.5     10.1.2.2          49505        445
10.1.2.5     10.1.2.3          49500        445
10.1.2.5     10.1.2.3          49510        445
10.1.2.5     10.1.2.3          49506        445
10.1.2.5     10.1.2.3          49503        445
10.1.2.5     10.1.2.4          49507        445
10.1.2.5     10.1.2.4          49502        445
10.1.2.5     10.1.2.4          49499        445
10.1.2.5     10.1.2.4          49504        445

Next, here’s the PowerShell script behind what was shown above. Here are a few notes on the script:

• It shows some text with a simplified view of the cmdlet about to be run, but the actual cmdlet is usually a bit more complex since it usually includes additional parameters and some formatting.
• I tried to use some color and extra blank lines so things would be easier to read.
• The “read-host” cmdlets are used to pause the script until I press enter to continue to the next phase.
• I pipe the results to “Out-Host” to make sure everything shows in proper order on the screen (without it, some of the longer running cmdlets might show out of order).
• The script is run from a client, so cmdlets that target the cluster use the –Cluster parameter and cmdlets that target a specific server use the –CimSession parameter.
• SQLIO is used as a workload generator. For more details, see this other blog post on using SQLIO.
• I make a point of using the PowerShell cmdlets for network information (Resolve-DnsName instead of NSLOOKUP.EXE, Get-NetTcpConnection instead of NETSTAT.EXE).

Cls

Write-Host -ForegroundColor Green "Cluster Information"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:\> Get-ClusterNode -Cluster JOSE-S"
Get-ClusterNode -Cluster JOSE-S | FT -AutoSize | Out-Host

Write-Host -ForegroundColor Yellow "PS C:\> Get-ClusterNetwork -Cluster JOSE-S"
Get-ClusterNetwork -Cluster JOSE-S | FT Cluster, Name, State, Address, Role -AutoSize | Out-Host

Write-Host -ForegroundColor Yellow "PS C:\> Get-ClusterResource -Cluster JOSE-S"
Get-ClusterResource -Cluster JOSE-S  | FT -AutoSize | Out-Host

Write-Host -ForegroundColor Yellow "PS C:\> Get-ClusterSharedVolume -Cluster JOSE-S"
Get-ClusterSharedVolume -Cluster JOSE-S  | FT -AutoSize | Out-Host

Read-Host

Write-Host -ForegroundColor Green "IP Addresses for each node, DNS for Scale-Out Name"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:\> Get-NetIPAddress -CimSession <node>"
"JOSE-S1", "JOSE-S2", "JOSE-S3" | % { Get-NetIPAddress -CimSession $_ -AddressFamily IPv4 -IPAddress 10.1.* | Sort IPAddress | FT PSComputerName, ifIndex, IPAddress, PrefixLength -AutoSize } | Out-Host

Write-Host -ForegroundColor Yellow "PS C:\> Resolve-DNSName JOSE-SO"
Resolve-DnsName JOSE-SO | FT -AutoSize | Out-Host

Read-Host

Write-Host -ForegroundColor Green "IP Addresses on each node per Scope name", ""
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:\> Get-SmbServerNetworkInterface -CimSession JOSE-SO"
1..3 | % { Get-SmbServerNetworkInterface -CimSession JOSE-S$_ | ? ScopeName -ne "*" | Sort ScopeName | FT PSComputerName, ScopeName, IPAddress -AutoSize }  | Out-Host

Read-Host

Write-Host -ForegroundColor Green "Simple connection just to check directory", ""
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:\> Dir \\JOSE-SO\<share>"
Dir \\JOSE-SO\Share1\testfile.dat, \\JOSE-SO\Share2\testfile.dat, \\JOSE-SO\Share3\testfile.dat  | Out-Host

Write-Host -ForegroundColor Green "SMB Connection", ""
Write-Host " "
Write-Host -ForegroundColor Yellow "PS C:\> Get-SmbConnection"
Get-SmbConnection | FT ServerName, ShareName, Dialect, Redirected, Credential -AutoSize  | Out-Host

Write-Host -ForegroundColor Green "TCP connections used by the client"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:\> Get-NetTCPConnection -RemotePort 445"
Get-NetTCPConnection -RemotePort 445 | Sort LocalAddress, RemoteAddress | FT LocalAddress, RemoteAddress, LocalPort, RemotePort -AutoSize  | Out-Host
Read-Host

Write-Host -ForegroundColor Green "Putting some load on the shares", ""
Write-Host " "
Write-Host -ForegroundColor Yellow "PS C:\> C:\SQLIO\SQLIO.EXE -s10 -kR -frandom -b8 -t8 -o16 -LS -BN <file>"
C:\SQLIO\SQLIO.EXE -s10 -kR -frandom -b8 -t8 -o16 -LS -BN \\JOSE-SO\Share1\testfile.dat \\JOSE-SO\Share2\testfile.dat \\JOSE-SO\Share3\testfile.dat

Write-Host -ForegroundColor Green "SMB Connection"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:\> Get-SmbConnection"
Get-SmbConnection | FT ServerName, ShareName, Dialect, Redirected, Credential -AutoSize | Out-Host

Write-Host -ForegroundColor Green "SMB Multichannel Connection"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:\> Get-SmbMultichannelConnection"
Get-SmbMultichannelConnection | Sort ClientIPAddress, ServerIPAddress | FT ClientIPAddress, ServerIPAddress, CurrentChannels | Out-Host

Write-Host -ForegroundColor Green "TCP connections used by the client"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:\> Get-NetTCPConnection -RemotePort 445"
Get-NetTCPConnection -RemotePort 445 | Sort LocalAddress, RemoteAddress | FT LocalAddress, RemoteAddress, LocalPort, RemotePort –AutoSize | Out-Host

You can find the slides and the recording for the presentation at http://channel9.msdn.com/Events/TechEd/NorthAmerica/2014/DCIM-B337

I recently published the Test-StorageHealth.ps1 script to the TechNet Script Center, which checks for the health and capacity of a Storage Cluster based on Windows Server 2012 R2 Scale-Out File Servers. This blog post explains the portion of the script that checks for Storage Cluster health, which is labeled “Phase 1”.

To assess health, the script basically looks at the few objects using PowerShell, making sure they are being reported as healthy. That includes a few cluster objects (nodes, networks, resources, CSV volumes) and some storage objects (volumes, virtual disks, storage pools, physical disks, enclosures). The script also checks related items like deduplicated volumes, continuously available SMB file shares, SMB open files and SMB witness connections. The script also checks a minimum number of items (4 enclosures, 240 physical disks, 3 pools, etc.) which you will likely have to adjust for your specific configuration.

Here’s a table of the objects, the properties that indicate health and the number of expected items included by default in the current version of the script (version 1.5):

It’s also important to note that, in general, the script looks at cluster objects and objects related to the scale-out file server, skipping local items. Here are a few examples:

• When checking storage objects, it filters for objects in the Cluster Storage Subsystem
• When checking volumes, it filters for volumes with the CSVFS File System
• When check SMB shares, it filter for those marked as continuously available

I hope you find the script and this blog post useful. You can use it directly, adjust a few items for your configuration or just use it as a starting point for creating your own custom script. You can download the script from the TechNet Script Center at http://gallery.technet.microsoft.com/scriptcenter/Test-StorageHealthps1-66d84fd4

I just published a PowerShell script to check the health and capacity of a Storage Cluster based on Windows Server 2012 R2 Scale-Out File Servers.

It assumes a specific configuration for Private Clouds using the solution described at http://technet.microsoft.com/en-us/library/dn554251.aspx.

It performs specific health checks for

• Failover Clustering (Cluster, Resources, Networks, Nodes)
• Storage Spaces (Physical Disks, Enclosures, Virtual Disks)
• Cluster Shared Volumes
• SMB File Shares
• Deduplication

It includes reporting of capacity by Pool, Volume and Deduplicated volume. It also includes collection of cluster logs and error events from all nodes in the cluster.

You can review and download the script from http://gallery.technet.microsoft.com/scriptcenter/Test-StorageHealthps1-66d84fd4

Lastly, if you're interested in how the script was written, you might also want to read these other blog posts:

• PowerShell tips for building objects with custom properties and special formatting
  http://blogs.technet.com/b/josebda/archive/2014/04/19/powershell-tips-for-building-objects-with-custom-properties-and-special-formatting.aspx
• How the Test-StorageHealth.ps1 script checks for Storage Cluster health
  http://blogs.technet.com/b/josebda/archive/2014/07/26/how-the-test-storagehealth-ps1-script-checks-for-storage-cluster-health.aspx

Introduction

Windows Server 2012 R2 introduced a new version of SMB. Technically it’s SMB version 3.02, but we continue to call it just SMB3. The main changes are described at http://technet.microsoft.com/en-us/library/hh831474.aspx.

With this new release, we made a few changes in SMB PowerShell to support the new scenarios and features.

This includes a few new cmdlets and some changes to existing cmdlets, with extra care not break any of your existing scripts.This blog post outlines one of the 7 set of changes related to SMB PowerShell in Windows Server 2012 R2.

Simpler setting of ACL for the folder behind a share

In Windows Server 2012, the SMB share has a property that facilitates applying the share ACL to the file system folder used by the share.

Here’s the syntax for a share named Share1:

• (Get-SmbShare –Name Share1 ).PresetPathACL | Set-Acl

In Windows Server 2012 R2, we have improved this scenario by providing a proper cmdlet to apply the share ACL to the file system used by the share.

Here’s the new syntax for the same share:

• Set-SmbPathAcl –ShareName Share1

Notes

1) The Windows Server 2012 syntax continues to work with Windows Server 2012 R2, but the new syntax is much simpler and therefore recommended.

2) There is known issue with Windows Server 2012 R2 Preview that causes this new cmdlet to fail when using non-Unicode languages. As a workaround, you can use the old syntax.

3) This blog post is an updated version of the September 2013 post at  http://blogs.technet.com/b/josebda/archive/2013/09/03/what-s-new-in-smb-powershell-in-windows-server-2012-r2.aspx focused on a single topic.

Introduction

Windows Server 2012 R2 introduced a new version of SMB. Technically it’s SMB version 3.02, but we continue to call it just SMB3. The main changes are described at http://technet.microsoft.com/en-us/library/hh831474.aspx.

With this new release, we made a few changes in SMB PowerShell to support the new scenarios and features. This includes a few new cmdlets and some changes to existing cmdlets, with extra care not break any of your existing scripts.

This blog post outlines one of the 7 set of changes related to SMB PowerShell in Windows Server 2012 R2.

Scale-Out Rebalancing (per-share redirection)

Per-share redirection is the new default behavior for the new SMB Clients (Windows 8.1 or Windows Server 2012 R2), when connecting to a Scale-Out clusters that use a storage system that does not support Direct I/O from all nodes, if running Windows Server 2012 R2. The common scenario here is a Scale-Out File Server backed by Mirrored Storage Spaces.

Here are the changes in SMB PowerShell to support SMB Scale-Out per-share redirection:

• New “Redirected” Boolean property to Get-SmbConnection to indicate that per-share redirection is being used.

• Get-SmbWitnessClient now includes a “ShareName” property, since witness can track connections per share, not only per server.

• Get-SmbWitnessClient now includes a “Flags” property, which will show “Share” when doing per-share redirection

Notes

1) For more details of the new rebalancing behavior of Windows Server 2012 R2, see http://blogs.technet.com/b/josebda/archive/2013/10/30/automatic-smb-scale-out-rebalancing-in-windows-server-2012-r2.aspx

2) This blog post is an updated version of the September 2013 post at  http://blogs.technet.com/b/josebda/archive/2013/09/03/what-s-new-in-smb-powershell-in-windows-server-2012-r2.aspx focused on a single topic.

Introduction

Windows Server 2012 R2 introduced a new version of SMB. Technically it’s SMB version 3.02, but we continue to call it just SMB3. The main changes are described at http://technet.microsoft.com/en-us/library/hh831474.aspx.

With this new release, we made a few changes in SMB PowerShell to support the new scenarios and features. This includes a few new cmdlets and some changes to existing cmdlets, with extra care not break any of your existing scripts.

This blog post outlines one of the 7 set of changes related to SMB PowerShell in Windows Server 2012 R2.

Changes related to SMB Witness

While the overall functionality of the SMB Witness is largely unchanged in Windows Server 2012 R2 (aside from per-share redirection), we have put some effort in improving the SMB PowerShell cmdlets associated with it. Here are the changes:

1) The Move-SmbWitnessClient cmdlet can be referred to now simply as Move-SmbClient. This is simply a new alias that better describes what the cmdlet actually does.

2) The default view for Get-SmbWitnessClient was improved. Here’s the new list of items shown:

Client Computer   Witness Node  File Server   Network Name  Share Name  Client State    
Name              Name             Node Name
----------------- ------------- ------------- ------------  ----------  ------------    
JOSE-V            JOSE-A1       JOSE-A2       JOSE-F        VMS4        RequestedNoti...
JOSE-V            JOSE-A1       JOSE-A2       JOSE-F        VMS3        RequestedNoti...
JOSE-V            JOSE-A1       JOSE-A3       JOSE-F        VMS1        RequestedNoti...
JOSE-V            JOSE-A1       JOSE-A3       JOSE-F        VMS2        RequestedNoti...
JOSE-V            JOSE-A2       JOSE-A1       JOSE-F        VMS6        RequestedNoti...
JOSE-V            JOSE-A2       JOSE-A1       JOSE-F        VMS5        RequestedNoti...

3) There is a new “NetworkName” parameter in Move-SmbClient.  If a NetworkName is specified, then only those will be moved.

Windows Server 2012 syntax for Move-SmbWitnessClient:

• Move-SmbWitnessClient -ClientName X -DestinationNode Y

Windows Server 2012 R2 syntax:

• Move-SmbClient -ClientName X -DestinationNode Y [ -NetworkName Z ]

Notes

1) If the –NetworkName is omitted in the Move-SmbClient cmdlet, all client connections will be moved to the destination.

2) When using per-share redirection, the SMB client will always move to the file server node that owns the volume behind the file share. Using Move-SmbClient (or Move-SmbWitnessClient) in that situation has no effect.

3) This blog post is an updated version of the September 2013 post at  http://blogs.technet.com/b/josebda/archive/2013/09/03/what-s-new-in-smb-powershell-in-windows-server-2012-r2.aspx focused on a single topic.

Introduction

Windows Server 2012 R2 introduced a new version of SMB. Technically it’s SMB version 3.02, but we continue to call it just SMB3. The main changes are described at http://technet.microsoft.com/en-us/library/hh831474.aspx.

With this new release, we made a few changes in SMB PowerShell to support the new scenarios and features. This includes a few new cmdlets and some changes to existing cmdlets, with extra care not break any of your existing scripts.

This blog post outlines one of the 7 set of changes related to SMB PowerShell in Windows Server 2012 R2.

SMB Bandwidth Limits

Starting with Windows Server 2012 R2, administrators can set a bandwidth limit for one or more categories by using simple PowerShell cmdlets. They can also query the limit for each category.  

There are 3 categories of traffic that can be limited:

• VirtualMachine: Hyper-V over SMB traffic. This limit can only be applied on Hyper-V hosts.

• LiveMigration: Hyper-V Live Migration over SMB traffic. This limit can only be applied on Hyper-V hosts.

• Default: All other types of SMB traffic. This limit can be applied on any SMB client.

A category limit is an absolute value expressed in bytes per second.  For example, a category limit of 500MB means the total throughput for this category should not exceed 500 megabytes per second.

Installing the new feature and performance counters

Before you can use Bandwidth Limits, the feature needs to be enabled.  For that, you should use the following PowerShell cmdlet:

• Add-WindowsFeature FS-SMBBW

A new SMB Performance counter set with an instance per category becomes available after you install the feature.  The performance counters in for this set will use the same counters used today for the SMB Client Shares counter.

New SmbBandwidthLimit cmdlets

The PowerShell cmdlets used to manage SMB Bandwidth Limits are:

• Get-SmbBandwidthLimit [ –Category {Default/VirtualMachine/LiveMigration} ]

• Set-SmbBandwidthLimit –Category {Default/VirtualMachine/LiveMigration} –BytesPerSecond x

• Remove-SmbBandwidthLimit –Category {Default/ VirtualMachine/LiveMigration}

PowerShell allows us to specify units like KB, MB, GB after the number of bytes when specifying the BytesPerSecond parameter.

The Set-SmbBandwidthLimit cmdlet won’t accept a BytesPerSecond parameter smaller than 1MB (1048576).

Note

This blog post is an updated version of the September 2013 post at  http://blogs.technet.com/b/josebda/archive/2013/09/03/what-s-new-in-smb-powershell-in-windows-server-2012-r2.aspx focused on a single topic.

Introduction

Windows Server 2012 R2 introduced a new version of SMB. Technically it’s SMB version 3.02, but we continue to call it just SMB3. The main changes are described at http://technet.microsoft.com/en-us/library/hh831474.aspx.

With this new release, we made a few changes in SMB PowerShell to support the new scenarios and features. This includes a few new cmdlets and some changes to existing cmdlets, with extra care not break any of your existing scripts.

This blog post outlines one of the 7 set of changes related to SMB PowerShell in Windows Server 2012 R2.

SMB Multi-instance

SMB Multi-instance is a new feature in Windows Server 2012 R2 that separates regular SMB traffic from CSV-related inter-node SMB traffic in distinct  SMB instances.

This is designed to improve isolation between the two types of traffic in improve the reliability of the SMB servers.

Information related to this new CSV-only instance in Windows Server 2012 R2 is typically hidden by default in all PowerShell cmdlets.

Showing hidden instance information

There are the changes in SMB PowerShell so an Administrator can view information related to the hidden CSV instance:

• The –IncludeHidden option in Get-SmbConnection and Get-SmbMultichannelConnection will show the connections associated with the hidden CSV instance.

• There is now an InstanceName property in the full output of Get-SmbConnection, Get-SmbMultichannelConnection, Get-SmbSession and Get-SmbOpenFile. It shows either “Default” or “CSV” (only shows if using the –IncludeHidden option).

There is really little use in inspecting the information on the hidden CSV instance, except if you’re troubleshooting CSV inter-node communications.

Note

This blog post is an updated version of the September 2013 post at  http://blogs.technet.com/b/josebda/archive/2013/09/03/what-s-new-in-smb-powershell-in-windows-server-2012-r2.aspx focused on a single topic.

Introduction

Windows Server 2012 R2 introduced a new version of SMB. Technically it’s SMB version 3.02, but we continue to call it just SMB3. The main changes are described at http://technet.microsoft.com/en-us/library/hh831474.aspx.

With this new release, we made a few changes in SMB PowerShell to support the new scenarios and features. This includes a few new cmdlets and some changes to existing cmdlets, with extra care not break any of your existing scripts.

This blog post outlines one of the 7 set of changes related to SMB PowerShell in Windows Server 2012 R2.

The need for SMB Delegation

When you configure Hyper-V over SMB and you manage your Hyper-V hosts remotely using Hyper-V Manager, you will might run into access denied messages. This is because you’re using your credentials from the remote machine running Hyper-V Manager in the Hyper-V host to access a third machine (the file server). This is what we call a “double-hop”, and it’s not allowed by default for security reasons.The main problem with the scenario is that an intruder that compromises one computer in your environment could then connect to other systems in your environments without the need to provide a username and password. One way to work around this issue is to connect directly to the Hyper-V host and providing your credentials at that time, avoiding the double-hop.

You can also address this by configuring Constrained Delegation for SMB shares, which is a process that involves changing properties in Active Directory. The security risk is reduced here because a potential intruder double-hop would be limited to that specific use case (using SMB shares on the specified servers). The constrained delegation process was greatly simplified in Windows Server 2012 when the the Active Directory team introduced resource-based Kerberos constrained delegation, as explained at http://technet.microsoft.com/library/hh831747.aspx. However, even with  this new resource-based constrained delegation, there are still quite a few steps to enable it.

Requirements for SMB Delegation

Before you use the new SMB Delegation cmdlets, you must meet two specific requirements.

First, the new cmdlets do rely on Active Directory PowerShell to perform their actions. For this reason, you need to install the Active Directory cmdlets before using the SMB delegation cmdlets. To install the Active Directory cmdlets, use:

• Install-WindowsFeature RSAT-AD-PowerShell

Second, these cmdlets rely on the the new resource-based delegation in Active Directory. Since that AD feature was introduced in Windows Server 2012, the Active Directory forest must be in “Windows Server 2012” functional level. To check the Active Directory Forest Functional level, use:

• Get-ADForest

The new SMB Delegation cmdlets

For Hyper-V over SMB in Windows Server 2012, we provided TechNet and blog-based guidance on how to automate constrained delegation. In Windows Server 2012 R2, SMB has a new set of cmdlets to simplify the configuration of resource-based constrained Delegation in SMB scenarios.

Here are the new cmdlets introduced:

• Get-SmbDelegation –SmbServer X

• Enable-SmbDelegation –SmbServer X –SmbClient Y

• Disable-SmbDelegation –SmbServer X [–SmbClient Y] [-Force]

Notes

1) For the Disable-SmbDelegation cmdlet, if no client is specified, delegation will be removed for all clients.

2) System Center Virtual Machine Manager uses a different method to remote into the Hyper-V host and configure SMB shares. When using VMM, constrained delegation is not required for management of Hyper-V of SMB.

3) This blog post is an updated version of the September 2013 post at  http://blogs.technet.com/b/josebda/archive/2013/09/03/what-s-new-in-smb-powershell-in-windows-server-2012-r2.aspx focused on a single topic.

Introduction

Windows Server 2012 R2 introduced a new version of SMB. Technically it’s SMB version 3.02, but we continue to call it just SMB3. The main changes are described at http://technet.microsoft.com/en-us/library/hh831474.aspx.

With this new release, we made a few changes in SMB PowerShell to support the new scenarios and features. This includes a few new cmdlets and some changes to existing cmdlets, with extra care not break any of your existing scripts.

This blog post outlines one of the 7 set of changes related to SMB PowerShell in Windows Server 2012 R2.

SMB1 can now be completely removed

In Windows Server 2012 R2, SMB1 became an optional component and can now be completely disabled, so that the associated binaries are not even loaded. For scenarios where SMB1 is not required, this means less resource utilization, less need for patching and improved security.

For instance, in the Hyper-V over SMB scenario, where you are storing Hyper-V virtual disks and virtual machine configuration in SMB file shares, SMB3 is a requirement. In this case, SMB1 is not necessary and can be safely disabled.

For information worker scenarios, if you have Windows XP clients, you absolutely still need SMB1, since that is the only SMB version supported by Windows XP. If *all* your clients are running Windows Vista or later, SMB1 is no longer required and you can disable SMB1. Windows Vista and Windows 7 do not need SMB1 since they support SMB2. Windows 8 and Windows 8.1 do not need SMB1, since they support both SMB2 and SMB3.

For classic server scenarios, if you have Windows Server 2003 or Windows Server 2003 R2 servers, you absolutely still need SMB1, since that is the only SMB version supported by them. Windows Server 2008 and Windows Server 2008 R2 do not need SMB1 since they support SMB2. Windows Server 2012 and Windows Server 2012 R2 do not need SMB1, since they support both SMB2 and SMB3.

Disable SMB1

Even though the component can now be removed, due to the compatibility issues listed above, it is still enabled by default.

To disable SMB1 completely, use the following PowerShell cmdlet:

• Remove-WindowsFeature FS-SMB1

You can re-enable it by using:

• Add-WindowsFeature FS-SMB1

Notes

1) A reboot is required after this feature is enabled or disabled.

2) For more details about SMB versions and dialects and which operating systems support them, see http://blogs.technet.com/b/josebda/archive/2013/10/02/windows-server-2012-r2-which-version-of-the-smb-protocol-smb-1-0-smb-2-0-smb-2-1-smb-3-0-or-smb-3-02-you-are-using.aspx

3) This blog post is an updated version of the September 2013 post at  http://blogs.technet.com/b/josebda/archive/2013/09/03/what-s-new-in-smb-powershell-in-windows-server-2012-r2.aspx focused on a single topic.

I recently published a PowerShell script to check the health and capacity of a Storage Cluster based on Windows Server 2012 R2 Scale-Out File Servers.

You can find details about this script (including the download link) in this blog post: PowerShell script for Storage Cluster Health Test published to the TechNet Script Center.

This script, when used without any parameters, simply performs a number of health checks and provides a report on health and capacity for the storage cluster.

However, when used with the optional –IncludeEvents parameter, it will collect lots of diagnostic information. The script also creates a convenient ZIP archive for transport.

Inside that ZIP (when you use the -IncludeEvents parameter), you will find a number of files that can help troubleshoot the storage cluster. That includes:

• System Information (txt file). You can open it with Notepad to review details about the system, including CPU, memory, network interfaces, OS version and much more.
• Cluster log (log file). This is a text file containing events from Failover Clustering. You can open it with Notepad to find every detail of what happened to the cluster.
• Event logs for Failover Clustering, Hyper-V Shared VHDX, SMB, Core Storage and Storage Spaces (evtx file). You can open these with Event Viewer to look at the specific events, with the usual options to filter and search.
• Mini dumps (dmp file). Dumps files contain information about what happened to the system during a crash. You can find more about how to read them at http://support.microsoft.com/kb/315263
• PowerShell objects (xml file). Use PowerShell to import these objects into memory and query them. For instance: Import-CliXml .\GetPhysicalDisk.XML | Select DeviceId, Model, FirmwareVersion

These files can be used by the Test-StorageHealth.ps1 script to recreate its output without being connected to the live cluster, using the –ReadFromPath parameter.

In addition to that, for a support professional or experienced administrator, these files include a wealth of information that can greatly help with troubleshooting.

Here is a comprehensive list of the files inside the ZIP created by the script:

A few weeks ago, I published a script to the TechNet Script Center that gathers health and capacity information for a Windows Server 2012 R2 storage cluster. This script checks a number of components, including clustering, storage spaces, SMB file shares and core storage. You can get more details at http://blogs.technet.com/b/josebda/archive/2014/07/27/powershell-script-for-storage-cluster-health-test-published-to-the-technet-script-center.aspx

From the beginning, I wanted to add the ability to look at storage performance in addition to health and capacity. This was added yesterday in version 1.7 of the script and this blog describes the decisions along the way and the details on how it was implemented.

Layers and objects

The first decision was the layer we should use to monitor storage performance. You can look at the core storage, storage spaces, cluster, SMB server, SMB client or Hyper-V. The last two would actually come from the client side, outside the storage cluster itself. Ideally I would gather storage performance from all these layers, but honestly that would be just too much to capture and also to review later.

We also have lots of objects, like physical disks, pools, tiers, virtual disks, partitions, volumes and file shares. Most people would agree that looking at volumes would be the most useful, if you had to choose only one. It helps that most deployments will use a single file share per volume and a single partition and volume per virtual disk. In these deployments, a share matches a single volume that matches a single virtual disk. Looking at pool (which typically hosts multiple virtual disks) would also be nice.

In the end, I decided to go with the view from the CSV file system, which looks at volumes (CSVs, really) and offers the metrics we need. It is also nice that it reports a consistent volume ID across the many nodes of the cluster. That made it easier to consolidate data and then correlate to the right volume, SMB share and pool.

Metrics

The other important decision is which of the many metrics we would show. There is plenty of information you can capture, including number of IOs, latency, bandwidth, queue depth and several others. These can also be captured as current values, total value since the system was started or an average for a given time period. In addition, you can capture those for read operations, write operations, metadata operations or all together.

After much debating with the team on this topic, we concluded the single most important metric to find out if a system is reaching its performance limit would be latency. IOs per second (IOPs) was also important to get an idea of how much work is currently being handled by the cluster. Differentiating read and write IOs would also be desirable, since they usually show different behavior.

To keep it reasonable, I made the call to gather read IOPS (average number of read operations per second), write IOPS, read latency (average latency of read operations per second, as measured by the CSV file system) and write latency. These 4 performance counters were captured per volume on every node of the cluster. Since it was easy and also useful, the script also shows total IOPS (sum of reads and writes) and total latency (average of reads and writes).

Samples

The other important decision was how many samples we would take. Ideally we would constantly monitor the storage performance and would have a full history of everything that ever happened to a storage cluster by the millisecond. That would be a lot of data, though. We would need another storage cluster just to store the performance information for a storage cluster :-) (just half joking here).

The other problem is that the storage health script aims to be as nonintrusive as possible. To constantly gather performance information we require need some sort of agent or service running on every node and we definitely did not want to go there.

The decision was to take a few samples only during the execution of the script. It gathers 60 samples, 1 second apart. During those 60 seconds the script is simply waiting and it's doing nothing else. I considered starting the capture on a separate thread (PowerShell job) and let it run while we’re gathering other health information, but I was afraid that the results would be impacted. I figured that waiting for one minute would be reasonable.

Capture

There are a few different ways to capture performance data. Using the performance counter infrastructure seems like the way to go, but even there you have a few different options. We could save the raw performance information to a file and parse it later. We could also use Win32 APIs to gather counters.

Since this is a PowerShell script, I decided to go with the Get-Counter cmdlet. It provides a simple way to get a specified list of counters, including multiple servers and multiple samples. The scripts uses a single command to gather all the relevant data, which is kept in memory and later processed.

Here’s some sample code:

$Samples = 60
$Nodes = “Cluster123N17”, “Cluster123N18”, “Cluster123N19”, “Cluster123N20”
$Names = “reads/sec”, “writes/sec” , “read latency”, “write latency”
$Items = $Nodes | % { $Node=$_; $Names | % { (”\\”+$Node+”\Cluster CSV File System(*)\”+$_) } }
$RawCounters = Get-Counter -Counter $Items -SampleInterval 1 -MaxSamples $Samples

The script then goes on to massage the data a bit. For the raw data, I wanted to fill in some related information (like pool and share) for every line. This would make it a proper fact table for Excel pivot tables, once it's exported to a comma-separated file. The other processing needed is summarizing the raw data into per-volume totals and averages.

Summary

I spent some time figuring out the best way to show a summary of the 60 seconds of performance data from the many nodes and volumes. The goal was to have something that would fit into a single screen for a typical configuration with a couple of dozen volumes.

The script shows one line per volume, but also includes the associated pool name and file share name. For each line you get read/write/total IOPS and also read/write/total latency. IOPS are shown as an integer and latency is shown in milliseconds with 3 decimals. The data is sorted in descending order by average latency, which should show the busiest volume/share on top.  

Here’s a sample output

Pool  Volume   Share    ReadIOPS WriteIOPS TotalIOPS ReadLatency (ms) WriteLatency (ms)
----  ------   -----    -------- --------- --------- ---------------- -----------------
Pool2 volume15 TShare8       162         6       168           33.771            52.893
Pool2 volume16 TShare9        38       858       896           37.241             17.12
Pool2 volume10 TShare11        0         9         9                0             6.749
Pool2 volume17 TShare10       20        19        39            4.128              8.95
Pool2 volume13 HShare         13       243       256            3.845             8.424
Pool2 volume11 TShare12        0         7         7            0.339             5.959
Pool1 volume8  TShare6       552       418       970            5.041             4.977
Pool2 volume14 TShare7         3        12        15            2.988             5.814
Pool3 volume28 BShare28        0        11        11                0             4.955
Pool1 volume6  TShare4       232         3       235            1.626             5.838
Pool1 volume7  TShare5        62       156       218            1.807             4.241
Pool1 volume3  TShare1         0         0         0                0                 0
Pool3 volume30 BShare30        0         0         0                0                 0
…

Excel

Another way to look at the data is to get the raw output and use Excel. You can find that data in a file is saved as a comma-separated values on the output folder (C:\HealthTest by default) under the name VolumePerformanceDetails.TXT.

If you know your way with Excel and pivot tables, you can extract more details. You have access to all the 60 samples and to the data for each of the nodes. The data also includes pool name, share name and owner node, which do not come with a simple Get-Counter cmdlet. Here is another example of a pivot table in Excel (using a different data set from the one shown above):

Conclusion

I hope you liked the new storage performance section of the Test-StorageHealth script. As with the rest of the script (on health and capacity) the idea is to provide simple way to get a useful summary and collect additional data you could dig into.

Let me know how it works for you. I welcome feedback on the specific choices made (layer, metrics, samples, summary) and further ideas on how to make it more useful.

In this post, I'm sharing my favorite links related to Storage Spaces in Windows Server 2012 R2. This includes TechEd Presentations, TechNet articles, Blogs and tools related to Storage Spaces in general and more specifically about its deployment in a Failover Cluster or Scale-Out File Server configuration. It's obviously not a complete reference (there are always new blogs and articles being posted), but hopefully this is a useful collection of links.

TechEd Presentations

• Storage Spaces: What’s New in Windows Server 2012 R2
• Best Practices for Deploying Tiered Storage Spaces in Windows Server 2012 R2

TechNet Articles – Storage Spaces

• Storage Spaces Overview
• Deploy Clustered Storage Spaces
• How Storage Spaces Makes Use of Hot Spares
• Monitor Storage Tiers Performance in Windows Server 2012 R2
• Replace Failed Disks and Repair JBODs for Storage Spaces in Windows Server
• Storage Cmdlets in Windows PowerShell

TechNet Wiki – Storage Spaces

• Storage Spaces Overview
• Storage Spaces Frequently Asked Questions (FAQ)
• Storage Spaces - Designing for Performance

TechNet Articles – Cost-Effective Storage for Hyper-V

• Provide cost-effective storage for Hyper-V workloads by using Windows Server
• Provide cost-effective storage for Hyper-V workloads by using Windows Server: planning and design guide

Blogs - Storage Spaces

• Step-by-step for Storage Spaces Tiering in Windows Server 2012 R2
• Step-by-Step for Mirrored Storage Spaces Resiliency using PowerShell
• SQLIO, PowerShell and storage performance: measuring IOPs, throughput and latency for both local disks and SMB file shares
• Using file copy to measure storage performance – Why it’s not a good idea and what you should do instead
• Storage Spaces: How to configure Storage Tiers with Windows Server 2012 R2

Script Center - Tools

• Test-StorageHealth Script– PowerShell script for gathering info on health, capacity, performance and events
• Storage Spaces Physical Disk Validation Script– PowerShell script for testing disks before deployment

Windows Server Catalog

• Hardware Certified for Storage Spaces

Let me know in the comments section if you find this useful or if you have additional links that you found useful.

1. Introduction

I have been doing storage-related demos and publishing blogs with some storage performance numbers for a while, and I commonly get questions such as “How do you run these tests?” or “What tools do you use to generate IOs for your demos?”. While it’s always best to use a real workload to test storage, sometimes that is not convenient. In the past, I frequently used and recommended a free tool from Microsoft to simulate IOs called SQLIO. However, there is a better tool that was recently released by Microsoft called DiskSpd. This is a flexible tool that can simulate many different types of workloads. And you can apply it to several configurations, from a physical host or virtual machine, using all kinds of storage, including local disks, LUNs on a SAN, Storage Spaces or SMB file shares.

2. Download the tool

To get started, you need to download and install the DiskSpd. You can get the tool from http://aka.ms/DiskSpd. It comes in the form of a ZIP file that you can open and copy local folder. There are actually 3 subfolders with different versions of the tool included in the ZIP file: amd64fre (for 64-bit systems), x86fre (for 32-bit systems) and armfre (for ARM systems). This allows you to run it in pretty much every Windows version, client or server.

In the end, you really only need one of the versions of DiskSpd.EXE files included in the ZIP (the one that best fits your platform). If you’re using a recent version of Windows Server, you probably want the version in the amd64fre folder. In this blog post, I assume that you copied the correct version of DiskSpd.EXE to the C:\DiskSpd local folder.

If you're a developer, you might also want to take a look at the source code for DiskSpd. You can find that at https://github.com/microsoft/diskspd.

3. Run the tool

When you’re ready to start running DiskSpd, you want to make sure there’s nothing else running on the computer. Other running process can interfere with your results by putting additional load on the CPU, network or storage. If the disk you are using is shared in any way (like a LUN on a SAN), you want to make sure that nothing else is competing with your testing. If you’re using any form of IP storage (iSCSI LUN, SMB file share), you want to make sure that you’re not running on a network congested with other kinds of traffic.

WARNING: You could be generating a whole lot of disk IO, network traffic and/or CPU load when you run DiskSpd. If you’re in a shared environment, you might want to talk to your administrator and ask permission. This could generate a whole lot of load and disturb anyone else using other VMs in the same host, other LUNs on the same SAN or other traffic on the same network.

WARNING: If you use DiskSpd to write data to a physical disk, you might destroy the data on that disk. DiskSpd does not ask for confirmation. It assumes you know what you are doing. Be careful when using physical disks (as opposed to files) with DiskSpd.

NOTE: You should run DiskSpd from an elevated command prompt. This will make sure file creation is fast. Otherwise, DiskSpd will fall back to a slower method of creating files. In the example below, when you're using a 1TB file, that might take a long time.

From an old command prompt or a PowerShell prompt, issue a single command line to start getting some performance results. Here is your first example using 8 threads of execution, each generating 8 outstanding random 8KB unbuffered read IOs:

PS C:\DiskSpd> C:\DiskSpd\diskspd.exe -c1000G -d10 -r -w0 -t8 -o8 -b8K -h -L X:\testfile.dat

Command Line: C:\DiskSpd\diskspd.exe -c1000G -d10 -r -w0 -t8 -o8 -b8K -h -L X:\testfile.dat

Input parameters:

        timespan:   1
        -------------
        duration: 10s
        warm up time: 5s
        cool down time: 0s
        measuring latency
        random seed: 0
        path: 'X:\testfile.dat'
                think time: 0ms
                burst size: 0
                software and hardware cache disabled
                performing read test
                block size: 8192
                using random I/O (alignment: 8192)
                number of outstanding I/O operations: 8
                stride size: 8192
                thread stride size: 0
                threads per file: 8
                using I/O Completion Ports
                IO priority: normal

Results for timespan 1:
*******************************************************************************

actual test time:       10.01s
thread count:           8
proc count:             4

CPU |  Usage |  User  |  Kernel |  Idle
-------------------------------------------
   0|   5.31%|   0.16%|    5.15%|  94.76%
   1|   1.87%|   0.47%|    1.40%|  98.19%
   2|   1.25%|   0.16%|    1.09%|  98.82%
   3|   2.97%|   0.47%|    2.50%|  97.10%
-------------------------------------------
avg.|   2.85%|   0.31%|    2.54%|  97.22%

Total IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |        20480000 |         2500 |       1.95 |     249.77 |   32.502 |    55.200 | X:\testfile.dat (1000GB)
     1 |        20635648 |         2519 |       1.97 |     251.67 |   32.146 |    54.405 | X:\testfile.dat (1000GB)
     2 |        21094400 |         2575 |       2.01 |     257.26 |   31.412 |    53.410 | X:\testfile.dat (1000GB)
     3 |        20553728 |         2509 |       1.96 |     250.67 |   32.343 |    56.548 | X:\testfile.dat (1000GB)
     4 |        20365312 |         2486 |       1.94 |     248.37 |   32.599 |    54.448 | X:\testfile.dat (1000GB)
     5 |        20160512 |         2461 |       1.92 |     245.87 |   32.982 |    54.838 | X:\testfile.dat (1000GB)
     6 |        19972096 |         2438 |       1.90 |     243.58 |   33.293 |    55.178 | X:\testfile.dat (1000GB)
     7 |        19578880 |         2390 |       1.87 |     238.78 |   33.848 |    58.472 | X:\testfile.dat (1000GB)
-----------------------------------------------------------------------------------------------------
total:         162840576 |        19878 |      15.52 |    1985.97 |   32.626 |    55.312

Read IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |        20480000 |         2500 |       1.95 |     249.77 |   32.502 |    55.200 | X:\testfile.dat (1000GB)
     1 |        20635648 |         2519 |       1.97 |     251.67 |   32.146 |    54.405 | X:\testfile.dat (1000GB)
     2 |        21094400 |         2575 |       2.01 |     257.26 |   31.412 |    53.410 | X:\testfile.dat (1000GB)
     3 |        20553728 |         2509 |       1.96 |     250.67 |   32.343 |    56.548 | X:\testfile.dat (1000GB)
     4 |        20365312 |         2486 |       1.94 |     248.37 |   32.599 |    54.448 | X:\testfile.dat (1000GB)
     5 |        20160512 |         2461 |       1.92 |     245.87 |   32.982 |    54.838 | X:\testfile.dat (1000GB)
     6 |        19972096 |         2438 |       1.90 |     243.58 |   33.293 |    55.178 | X:\testfile.dat (1000GB)
     7 |        19578880 |         2390 |       1.87 |     238.78 |   33.848 |    58.472 | X:\testfile.dat (1000GB)
-----------------------------------------------------------------------------------------------------
total:         162840576 |        19878 |      15.52 |    1985.97 |   32.626 |    55.312

Write IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | X:\testfile.dat (1000GB)
     1 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | X:\testfile.dat (1000GB)
     2 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | X:\testfile.dat (1000GB)
     3 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | X:\testfile.dat (1000GB)
     4 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | X:\testfile.dat (1000GB)
     5 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | X:\testfile.dat (1000GB)
     6 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | X:\testfile.dat (1000GB)
     7 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | X:\testfile.dat (1000GB)
-----------------------------------------------------------------------------------------------------
total:                 0 |            0 |       0.00 |       0.00 |    0.000 |       N/A

  %-ile |  Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
    min |      3.360 |        N/A |      3.360
   25th |      5.031 |        N/A |      5.031
   50th |      8.309 |        N/A |      8.309
   75th |     12.630 |        N/A |     12.630
   90th |    148.845 |        N/A |    148.845
   95th |    160.892 |        N/A |    160.892
   99th |    172.259 |        N/A |    172.259
3-nines |    254.020 |        N/A |    254.020
4-nines |    613.602 |        N/A |    613.602
5-nines |    823.760 |        N/A |    823.760
6-nines |    823.760 |        N/A |    823.760
7-nines |    823.760 |        N/A |    823.760
8-nines |    823.760 |        N/A |    823.760
    max |    823.760 |        N/A |    823.760

NOTE: The -w0 is the default, so you could skip it. I'm keeping it here to be explicit about the fact we're doing all reads.

For this specific disk, I am getting 1,985 IOPS, 15.52 MB/sec of average throughput and 32.626 milliseconds of average latency. I’m getting all that information from the blue line above.

That average latency looks high for small IOs (even though this is coming from a set of HDDs), but we’ll examine that later.

Now, let’s try now another command using sequential 512KB reads on that same file. I’ll use 2 threads with 8 outstanding IOs per thread this time:

PS C:\DiskSpd> C:\DiskSpd\diskspd.exe -c1000G -d10 -w0 -t2 -o8 -b512K -h -L X:\testfile.dat

Command Line: C:\DiskSpd\diskspd.exe -c1000G -d10 -w0 -t2 -o8 -b512K -h -L X:\testfile.dat

Input parameters:

        timespan:   1
        -------------
        duration: 10s
        warm up time: 5s
        cool down time: 0s
        measuring latency
        random seed: 0
        path: 'X:\testfile.dat'
                think time: 0ms
                burst size: 0
                software and hardware cache disabled
                performing read test
                block size: 524288
                number of outstanding I/O operations: 8
                stride size: 524288
                thread stride size: 0
                threads per file: 2
                using I/O Completion Ports
                IO priority: normal

Results for timespan 1:
*******************************************************************************

actual test time:       10.00s
thread count:           2
proc count:             4

CPU |  Usage |  User  |  Kernel |  Idle
-------------------------------------------
   0|   4.53%|   0.31%|    4.22%|  95.44%
   1|   1.25%|   0.16%|    1.09%|  98.72%
   2|   0.00%|   0.00%|    0.00%|  99.97%
   3|   0.00%|   0.00%|    0.00%|  99.97%
-------------------------------------------
avg.|   1.44%|   0.12%|    1.33%|  98.52%

Total IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |       886046720 |         1690 |      84.47 |     168.95 |   46.749 |    47.545 | X:\testfile.dat (1000GB)
     1 |       851443712 |         1624 |      81.17 |     162.35 |   49.497 |    54.084 | X:\testfile.dat (1000GB)
-----------------------------------------------------------------------------------------------------
total:        1737490432 |         3314 |     165.65 |     331.29 |   48.095 |    50.873

Read IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |       886046720 |         1690 |      84.47 |     168.95 |   46.749 |    47.545 | X:\testfile.dat (1000GB)
     1 |       851443712 |         1624 |      81.17 |     162.35 |   49.497 |    54.084 | X:\testfile.dat (1000GB)
-----------------------------------------------------------------------------------------------------
total:        1737490432 |         3314 |     165.65 |     331.29 |   48.095 |    50.873

Write IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | X:\testfile.dat (1000GB)
     1 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | X:\testfile.dat (1000GB)
-----------------------------------------------------------------------------------------------------
total:                 0 |            0 |       0.00 |       0.00 |    0.000 |       N/A

  %-ile |  Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
    min |      9.406 |        N/A |      9.406
   25th |     31.087 |        N/A |     31.087
   50th |     38.397 |        N/A |     38.397
   75th |     47.216 |        N/A |     47.216
   90th |     64.783 |        N/A |     64.783
   95th |     90.786 |        N/A |     90.786
   99th |    356.669 |        N/A |    356.669
3-nines |    452.198 |        N/A |    452.198
4-nines |    686.307 |        N/A |    686.307
5-nines |    686.307 |        N/A |    686.307
6-nines |    686.307 |        N/A |    686.307
7-nines |    686.307 |        N/A |    686.307
8-nines |    686.307 |        N/A |    686.307
    max |    686.307 |        N/A |    686.307

With that configuration and parameters, I got about 165.65 MB/sec of throughput with an average latency of 48.095 milliseconds per IO. Again, that latency sounds high even for 512KB IOs and we’ll dive into that topic later on.

5. Understand the parameters used

Now let’s inspect the parameters on those DiskSpd command lines. I know it’s a bit overwhelming at first, but you will get used to it. And keep in mind that, for DiskSpd parameters, lowercase and uppercase mean different things, so be very careful.

Here is the explanation for the parameters used above:

PS C:\> C:\DiskSpd\diskspd.exe -c1G -d10 -r -w0 -t8 -o8 -b8K -h -L X:\testfile.dat

For OLTP workloads, I commonly start with 8KB random IOs, 8 threads, 16 outstanding per thread. 8KB is the size of the page used by SQL Server for its data files. In parameter form, that would be: –r -b8K -t8 -o16. For reporting or OLAP workloads with large IO, I commonly start with 512KB IOs, 2 threads and 16 outstanding per thread. 512KB is a common IO size when SQL Server loads a batch of 64 data pages when using the read ahead technique for a table scan. In parameter form, that would be: -b512K -t2 -o16. These numbers will need to be adjusted if your machine has many cores and/or if you volume is backed up by a large number of physical disks.

If you’re curious, here are more details about parameters for DiskSpd, coming from the tool’s help itself:

PS C:\> C:\DiskSpd\diskspd.exe

Usage: C:\DiskSpd\diskspd.exe [options] target1 [ target2 [ target3 ...] ]
version 2.0.12 (2014/09/17)

Available targets:
       file_path
       #<physical drive number>
       <partition_drive_letter>:

Available options:
  -?                 display usage information
  -a#[,#[...]]       advanced CPU affinity - affinitize threads to CPUs provided after -a
                       in a round-robin manner within current KGroup (CPU count starts with 0); the same CPU
                       can be listed more than once and the number of CPUs can be different
                       than the number of files or threads (cannot be used with -n)
  -ag                group affinity - affinitize threads in a round-robin manner across KGroups
  -b<size>[K|M|G]    block size in bytes/KB/MB/GB [default=64K]
  -B<offs>[K|M|G|b]  base file offset in bytes/KB/MB/GB/blocks [default=0]
                       (offset from the beginning of the file)
  -c<size>[K|M|G|b]  create files of the given size.
                       Size can be stated in bytes/KB/MB/GB/blocks
  -C<seconds>        cool down time - duration of the test after measurements finished [default=0s].
  -D<bucketDuration> Print IOPS standard deviations. The deviations are calculated for samples of duration <bucketDuration>.
                       <bucketDuration> is given in milliseconds and the default value is 1000.
  -d<seconds>        duration (in seconds) to run test [default=10s]
  -f<size>[K|M|G|b]  file size - this parameter can be used to use only the part of the file/disk/partition
                       for example to test only the first sectors of disk
  -fr                open file with the FILE_FLAG_RANDOM_ACCESS hint
  -fs                open file with the FILE_FLAG_SEQUENTIAL_SCAN hint
  -F<count>          total number of threads (cannot be used with -t)
  -g<bytes per ms>   throughput per thread is throttled to given bytes per millisecond
                       note that this can not be specified when using completion routines
  -h                 disable both software and hardware caching
  -i<count>          number of IOs (burst size) before thinking. must be specified with -j
  -j<duration>       time to think in ms before issuing a burst of IOs (burst size). must be specified with -i
  -I<priority>       Set IO priority to <priority>. Available values are: 1-very low, 2-low, 3-normal (default)
  -l                 Use large pages for IO buffers
  -L                 measure latency statistics
  -n                 disable affinity (cannot be used with -a)
  -o<count>          number of overlapped I/O requests per file per thread
                       (1=synchronous I/O, unless more than 1 thread is specified with -F)
                       [default=2]
  -p                 start async (overlapped) I/O operations with the same offset
                       (makes sense only with -o2 or grater)
  -P<count>          enable printing a progress dot after each <count> completed I/O operations
                       (counted separately by each thread) [default count=65536]
  -r<align>[K|M|G|b] random I/O aligned to <align> bytes (doesn't make sense with -s).
                       <align> can be stated in bytes/KB/MB/GB/blocks
                       [default access=sequential, default alignment=block size]
  -R<text|xml>       output format. Default is text.
  -s<size>[K|M|G|b]  stride size (offset between starting positions of subsequent I/O operations)
  -S                 disable OS caching
  -t<count>          number of threads per file (cannot be used with -F)
  -T<offs>[K|M|G|b]  stride between I/O operations performed on the same file by different threads
                       [default=0] (starting offset = base file offset + (thread number * <offs>)
                       it makes sense only with -t or -F
  -v                 verbose mode
  -w<percentage>     percentage of write requests (-w and -w0 are equivalent).
                     absence of this switch indicates 100% reads
                       IMPORTANT: Your data will be destroyed without a warning
  -W<seconds>        warm up time - duration of the test before measurements start [default=5s].
  -x                 use completion routines instead of I/O Completion Ports
  -X<path>           use an XML file for configuring the workload. Cannot be used with other parameters.
  -z                 set random seed [default=0 if parameter not provided, GetTickCount() if value not provided]

Write buffers:
  -Z                        zero buffers used for write tests
  -Z<size>[K|M|G|b]         use a global <size> buffer filled with random data as a source for write operations.
  -Z<size>[K|M|G|b],<file>  use a global <size> buffer filled with data from <file> as a source for write operations.
                              If <file> is smaller than <size>, its content will be repeated multiple times in the buffer.

  By default, the write buffers are filled with a repeating pattern (0, 1, 2, ..., 255, 0, 1, ...)

Synchronization:
  -ys<eventname>     signals event <eventname> before starting the actual run (no warmup)
                       (creates a notification event if <eventname> does not exist)
  -yf<eventname>     signals event <eventname> after the actual run finishes (no cooldown)
                       (creates a notification event if <eventname> does not exist)
  -yr<eventname>     waits on event <eventname> before starting the run (including warmup)
                       (creates a notification event if <eventname> does not exist)
  -yp<eventname>     allows to stop the run when event <eventname> is set; it also binds CTRL+C to this event
                       (creates a notification event if <eventname> does not exist)
  -ye<eventname>     sets event <eventname> and quits

Event Tracing:
  -ep                   use paged memory for NT Kernel Logger (by default it uses non-paged memory)
  -eq                   use perf timer
  -es                   use system timer (default)
  -ec                   use cycle count
  -ePROCESS             process start & end
  -eTHREAD              thread start & end
  -eIMAGE_LOAD          image load
  -eDISK_IO             physical disk IO
  -eMEMORY_PAGE_FAULTS  all page faults
  -eMEMORY_HARD_FAULTS  hard faults only
  -eNETWORK             TCP/IP, UDP/IP send & receive
  -eREGISTRY            registry calls

Examples:

Create 8192KB file and run read test on it for 1 second:

  C:\DiskSpd\diskspd.exe -c8192K -d1 testfile.dat

Set block size to 4KB, create 2 threads per file, 32 overlapped (outstanding)
I/O operations per thread, disable all caching mechanisms and run block-aligned random
access read test lasting 10 seconds:

  C:\DiskSpd\diskspd.exe -b4K -t2 -r -o32 -d10 -h testfile.dat

Create two 1GB files, set block size to 4KB, create 2 threads per file, affinitize threads
to CPUs 0 and 1 (each file will have threads affinitized to both CPUs) and run read test
lasting 10 seconds:

  C:\DiskSpd\diskspd.exe -c1G -b4K -t2 -d10 -a0,1 testfile1.dat testfile2.dat

6. Tune the parameters for large sequential IO

Now that you have the basics down, we can spend some time looking at how you can refine your number of threads and queue depth for your specific configuration. This might help us figure out why we had those higher than expected latency numbers in the initial runs. You basically need to experiment with the -t and the -o parameters until you find the one that give you the best results. You first want to find out the latency for a given system with a queue depth of 1. Then you can increase the queue depth and check what happens in terms of IOPs, throughput and latency.

Keep in mind that many logical (and “physical”) disks may have multiple IO paths.  That’s the case in the examples mentioned here, but also true for most cloud storage systems and some physical drives (especially SSDs).  In general, increasing outstanding IOs will have minimal impact on latency until the IO paths start to saturate. Then latency will start to increase dramatically.

Here’s a sample script that measures queue depth from 1 to 16, parsing the output of DiskSpd to give us just the information we need. The results for each DiskSpd run are stored in the $result variable and parsed to show IOPs, throughput, latency and CPU usage on a single line. There is some fun string parsing going on there, first to find the line that contains the information we’re looking for, and then using the Split() function to break that line into the individual metrics we need. DiskSpd has the -Rxml option to output XML instead of text, but for me it was easier to parse the text.

1..16 | % { 
   $param = "-o $_"
   $result = C:\DiskSpd\diskspd.exe -c1000G -d10 -w0 -t1 $param -b512K -h -L X:\testfile.dat
   foreach ($line in $result) {if ($line -like "total:*") { $total=$line; break } }
   foreach ($line in $result) {if ($line -like "avg.*") { $avg=$line; break } }
   $mbps = $total.Split("|")[2].Trim()
   $iops = $total.Split("|")[3].Trim()
   $latency = $total.Split("|")[4].Trim()
   $cpu = $avg.Split("|")[1].Trim()
   "Param $param, $iops iops, $mbps MB/sec, $latency ms, $cpu CPU"
}

Here is the output:

Param -o 1, 61.01 iops, 30.50 MB/sec, 16.355 ms, 0.20% CPU
Param -o 2, 140.99 iops, 70.50 MB/sec, 14.143 ms, 0.31% CPU
Param -o 3, 189.00 iops, 94.50 MB/sec, 15.855 ms, 0.47% CPU
Param -o 4, 248.20 iops, 124.10 MB/sec, 16.095 ms, 0.47% CPU
Param -o 5, 286.45 iops, 143.23 MB/sec, 17.431 ms, 0.94% CPU
Param -o 6, 316.05 iops, 158.02 MB/sec, 19.052 ms, 0.78% CPU
Param -o 7, 332.51 iops, 166.25 MB/sec, 21.059 ms, 0.66% CPU
Param -o 8, 336.16 iops, 168.08 MB/sec, 23.875 ms, 0.82% CPU
Param -o 9, 339.95 iops, 169.97 MB/sec, 26.482 ms, 0.55% CPU
Param -o 10, 340.93 iops, 170.46 MB/sec, 29.373 ms, 0.70% CPU
Param -o 11, 338.58 iops, 169.29 MB/sec, 32.567 ms, 0.55% CPU
Param -o 12, 344.98 iops, 172.49 MB/sec, 34.675 ms, 1.09% CPU
Param -o 13, 332.09 iops, 166.05 MB/sec, 39.190 ms, 0.82% CPU
Param -o 14, 341.05 iops, 170.52 MB/sec, 41.127 ms, 1.02% CPU
Param -o 15, 339.73 iops, 169.86 MB/sec, 44.037 ms, 0.39% CPU
Param -o 16, 335.43 iops, 167.72 MB/sec, 47.594 ms, 0.86% CPU

For large sequential IOs, we typically want to watch the throughput (in MB/sec). There is a significant increase until we reach 6 outstanding IOs, which gives us around 158 MB/sec with 19 millisecond of latency per IO. You can clearly see that if you don’t queue up some IO, you’re not extracting the full throughput of this disk, since we’ll be processing the data while the disks are idle waiting for more work. If we queue more than 6 IOs, we really don’t get much more throughput, we only manage to increase the latency, as the disk subsystem is unable to give you much more throughput. You can queue up 10 IOs to reach 170 MB/sec, but we increase latency to nearly 30 milliseconds (a latency increase of 50% for a gain of only 8% in throughput).

At this point, it is clear that using multiple outstanding IOs is a great idea. However, using more outstanding IOs than what your target application can drive will be misleading as it will achieve throughput the application isn’t architected to achieve.  Using less outstanding IOs than what the application can drive may lead to an incorrect conclusion that the disk can’t achieve the necessary throughput, because the full parallelism of the disk isn’t being utilized. You should try to find what your specific application does to make sure that your DiskSpd simulation is a good approximation of your real workload.

So, looking at the data above, we can conclude that 6 outstanding IOs is a reasonable number for this storage subsystem. Now we can see if we can gain by spreading the work across multiple threads. What we want to avoid here is bottlenecking on a single CPU core, which is very common we doing lots and lots of IO. A simple experiment is to double the number of threads while reducing the queue depth by half.  Let’s now try 2 threads instead of 1.

1..8 | % {  
   $param = "-o $_"
   $result = C:\DiskSpd\diskspd.exe -c1000G -d10 -w0 -t2 $param -b512K -h -L X:\testfile.dat
   foreach ($line in $result) {if ($line -like "total:*") { $total=$line; break } }
   foreach ($line in $result) {if ($line -like "avg.*") { $avg=$line; break } }
   $mbps = $total.Split("|")[2].Trim() 
   $iops = $total.Split("|")[3].Trim()
   $latency = $total.Split("|")[4].Trim()
   $cpu = $avg.Split("|")[1].Trim()
   "Param –t2 $param, $iops iops, $mbps MB/sec, $latency ms, $cpu CPU"
}

Here is the output with 2 threads and a queue depth of 1:

Param –t2 -o 1, 162.01 iops, 81.01 MB/sec, 12.500 ms, 0.35% CPU
Param –t2 -o 2, 250.47 iops, 125.24 MB/sec, 15.956 ms, 0.82% CPU
Param –t2 -o 3, 312.52 iops, 156.26 MB/sec, 19.137 ms, 0.98% CPU
Param –t2 -o 4, 331.28 iops, 165.64 MB/sec, 24.136 ms, 0.82% CPU
Param –t2 -o 5, 342.45 iops, 171.23 MB/sec, 29.180 ms, 0.74% CPU
Param –t2 -o 6, 340.59 iops, 170.30 MB/sec, 35.391 ms, 1.17% CPU
Param –t2 -o 7, 337.75 iops, 168.87 MB/sec, 41.400 ms, 1.05% CPU
Param –t2 -o 8, 336.15 iops, 168.08 MB/sec, 47.859 ms, 0.90% CPU

Well, it seems like we were not bottlenecked on CPU after all (we sort of knew that already). So, with 2 threads and 3 outstanding IOs per thread, we effective get 6 total outstanding IOs and the performance numbers match what we got with 1 thread and queue depth of 6 in terms of throughput and latency. That pretty much proves that 1 thread was enough for this kind of configuration and workload and that increasing the number of threads yields no gain. This is not surprising for large IO. However, for smaller IO size, the CPU is more taxed and we might hit a single core bottleneck. We can look at the full DiskSpd output to confirm that no single core has pegged with 1 thread:

PS C:\DiskSpd> C:\DiskSpd\diskspd.exe -c1000G -d10 -w0 -t1 -o6 -b512K -h -L X:\testfile.dat

Command Line: C:\DiskSpd\diskspd.exe -c1000G -d10 -w0 -t1 -o6 -b512K -h -L X:\testfile.dat

Input parameters:

        timespan:   1
        -------------
        duration: 10s
        warm up time: 5s
        cool down time: 0s
        measuring latency
        random seed: 0
        path: 'X:\testfile.dat'
                think time: 0ms
                burst size: 0
                software and hardware cache disabled
                performing read test
                block size: 524288
                number of outstanding I/O operations: 6
                stride size: 524288
                thread stride size: 0
                threads per file: 1
                using I/O Completion Ports
                IO priority: normal

Results for timespan 1:
*******************************************************************************

actual test time:       10.00s
thread count:           1
proc count:             4

CPU |  Usage |  User  |  Kernel |  Idle
-------------------------------------------
   0|   2.03%|   0.16%|    1.87%|  97.96%
   1|   0.00%|   0.00%|    0.00%|  99.84%
   2|   0.00%|   0.00%|    0.00%| 100.15%
   3|   0.00%|   0.00%|    0.00%| 100.31%
-------------------------------------------
avg.|   0.51%|   0.04%|    0.47%|  99.56%

Total IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |      1664614400 |         3175 |     158.74 |     317.48 |   18.853 |    21.943 | X:\testfile.dat (1000GB)
-----------------------------------------------------------------------------------------------------
total:        1664614400 |         3175 |     158.74 |     317.48 |   18.853 |    21.943

Read IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |      1664614400 |         3175 |     158.74 |     317.48 |   18.853 |    21.943 | X:\testfile.dat (1000GB)
-----------------------------------------------------------------------------------------------------
total:        1664614400 |         3175 |     158.74 |     317.48 |   18.853 |    21.943

Write IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | X:\testfile.dat (1000GB)
-----------------------------------------------------------------------------------------------------
total:                 0 |            0 |       0.00 |       0.00 |    0.000 |       N/A

  %-ile |  Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
    min |      7.743 |        N/A |      7.743
   25th |     13.151 |        N/A |     13.151
   50th |     15.301 |        N/A |     15.301
   75th |     17.777 |        N/A |     17.777
   90th |     22.027 |        N/A |     22.027
   95th |     29.791 |        N/A |     29.791
   99th |    102.261 |        N/A |    102.261
3-nines |    346.305 |        N/A |    346.305
4-nines |    437.603 |        N/A |    437.603
5-nines |    437.603 |        N/A |    437.603
6-nines |    437.603 |        N/A |    437.603
7-nines |    437.603 |        N/A |    437.603
8-nines |    437.603 |        N/A |    437.603
    max |    437.603 |        N/A |    437.603

This confirms we’re not bottleneck on any of CPU cores. You can see above that the busiest CPU core is at only around 2% use.

7. Tune queue depth for small random IOs

Performing the same tuning exercise for small random IOS is typically more interesting, especially when you have fast storage. For this one, we’ll continue to use the same PowerShell script. However, for small IOs, we’ll try a larger number for queue depth. This might take a while to run, though… Here’s a script that you can run from a PowerShell prompt, trying out many different queue depths:

1..24 | % { 
   $param = "-o $_"
   $result = C:\DiskSpd\DiskSpd.exe -c1000G -d10 -w0 -r -b8k $param -t1 -h -L X:\testfile.dat
   foreach ($line in $result) {if ($line -like "total:*") { $total=$line; break } }
   foreach ($line in $result) {if ($line -like "avg.*") { $avg=$line; break } }
   $mbps = $total.Split("|")[2].Trim()
   $iops = $total.Split("|")[3].Trim()
   $latency = $total.Split("|")[4].Trim()
   $cpu = $avg.Split("|")[1].Trim()  
   "Param $param, $iops iops, $mbps MB/sec, $latency ms, $cpu CPU"
}

As you can see, the  script runs DiskSpd 24 times, using different queue depths. Here’s the sample output:

Param -o 1, 191.06 iops, 1.49 MB/sec, 5.222 ms, 0.27% CPU
Param -o 2, 361.10 iops, 2.82 MB/sec, 5.530 ms, 0.82% CPU
Param -o 3, 627.30 iops, 4.90 MB/sec, 4.737 ms, 1.02% CPU
Param -o 4, 773.70 iops, 6.04 MB/sec, 5.164 ms, 1.02% CPU
Param -o 5, 1030.65 iops, 8.05 MB/sec, 4.840 ms, 0.86% CPU
Param -o 6, 1191.29 iops, 9.31 MB/sec, 5.030 ms, 1.33% CPU
Param -o 7, 1357.42 iops, 10.60 MB/sec, 5.152 ms, 1.13% CPU
Param -o 8, 1674.22 iops, 13.08 MB/sec, 4.778 ms, 2.07% CPU
Param -o 9, 1895.25 iops, 14.81 MB/sec, 4.745 ms, 1.60% CPU
Param -o 10, 2097.54 iops, 16.39 MB/sec, 4.768 ms, 1.95% CPU
Param -o 11, 2014.49 iops, 15.74 MB/sec, 5.467 ms, 2.03% CPU
Param -o 12, 1981.64 iops, 15.48 MB/sec, 6.055 ms, 1.84% CPU
Param -o 13, 2000.11 iops, 15.63 MB/sec, 6.517 ms, 1.72% CPU
Param -o 14, 1968.79 iops, 15.38 MB/sec, 7.113 ms, 1.79% CPU
Param -o 15, 1970.69 iops, 15.40 MB/sec, 7.646 ms, 2.34% CPU
Param -o 16, 1983.77 iops, 15.50 MB/sec, 8.069 ms, 1.80% CPU
Param -o 17, 1976.84 iops, 15.44 MB/sec, 8.599 ms, 1.56% CPU
Param -o 18, 1982.57 iops, 15.49 MB/sec, 9.049 ms, 2.11% CPU
Param -o 19, 1993.13 iops, 15.57 MB/sec, 9.577 ms, 2.30% CPU
Param -o 20, 1967.71 iops, 15.37 MB/sec, 10.121 ms, 2.30% CPU
Param -o 21, 1964.76 iops, 15.35 MB/sec, 10.699 ms, 1.29% CPU
Param -o 22, 1984.55 iops, 15.50 MB/sec, 11.099 ms, 1.76% CPU
Param -o 23, 1965.34 iops, 15.35 MB/sec, 11.658 ms, 1.37% CPU
Param -o 24, 1983.87 iops, 15.50 MB/sec, 12.161 ms, 1.48% CPU

As you can see, for small IOs, we got consistently better performance as we increased the queue depth for the first few runs. After a certain number of outstanding IOs, adding more started giving us very little improvement until things flatten out completely. As we kept adding more queue depth, all we had was more latency with no additional benefit in IOPS or throughput. If you have a better storage subsystem, you might need to try even higher queue depths. If you don’t hit an IOPS plateau with increasing average latency, you did not queue enough IO to fully exploit the capabilities of your storage subsystem.

So, in this setup, we seem to reach a limit at around 10 outstanding IOs and latency starts to ramp up more dramatically after that. Let’s see the full output for queue depth of 10 to get a good sense:

PS C:\DiskSpd> C:\DiskSpd\DiskSpd.exe -c1000G -d10 -w0 -r -b8k -o10 -t1 -h -L X:\testfile.dat

Command Line: C:\DiskSpd\DiskSpd.exe -c1000G -d10 -w0 -r -b8k -o10 -t1 -h -L X:\testfile.dat

Input parameters:

        timespan:   1
        -------------
        duration: 10s
        warm up time: 5s
        cool down time: 0s
        measuring latency
        random seed: 0
        path: 'X:\testfile.dat'
                think time: 0ms
                burst size: 0
                software and hardware cache disabled
                performing read test
                block size: 8192
                using random I/O (alignment: 8192)
                number of outstanding I/O operations: 10
                stride size: 8192
                thread stride size: 0
                threads per file: 1
                using I/O Completion Ports
                IO priority: normal

Results for timespan 1:
*******************************************************************************

actual test time:       10.01s
thread count:           1
proc count:             4

CPU |  Usage |  User  |  Kernel |  Idle
-------------------------------------------
   0|   8.58%|   1.09%|    7.49%|  91.45%
   1|   0.00%|   0.00%|    0.00%| 100.03%
   2|   0.00%|   0.00%|    0.00%|  99.88%
   3|   0.00%|   0.00%|    0.00%| 100.03%
-------------------------------------------
avg.|   2.15%|   0.27%|    1.87%|  97.85%

Total IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |       160145408 |        19549 |      15.25 |    1952.47 |    5.125 |     8.135 | X:\testfile.dat (1000GB)
-----------------------------------------------------------------------------------------------------
total:         160145408 |        19549 |      15.25 |    1952.47 |    5.125 |     8.135

Read IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |       160145408 |        19549 |      15.25 |    1952.47 |    5.125 |     8.135 | X:\testfile.dat (1000GB)
-----------------------------------------------------------------------------------------------------
total:         160145408 |        19549 |      15.25 |    1952.47 |    5.125 |     8.135

Write IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | X:\testfile.dat (1000GB)
-----------------------------------------------------------------------------------------------------
total:                 0 |            0 |       0.00 |       0.00 |    0.000 |       N/A

  %-ile |  Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
    min |      3.101 |        N/A |      3.101
   25th |      3.961 |        N/A |      3.961
   50th |      4.223 |        N/A |      4.223
   75th |      4.665 |        N/A |      4.665
   90th |      5.405 |        N/A |      5.405
   95th |      6.681 |        N/A |      6.681
   99th |     21.494 |        N/A |     21.494
3-nines |    123.648 |        N/A |    123.648
4-nines |    335.632 |        N/A |    335.632
5-nines |    454.760 |        N/A |    454.760
6-nines |    454.760 |        N/A |    454.760
7-nines |    454.760 |        N/A |    454.760
8-nines |    454.760 |        N/A |    454.760
    max |    454.760 |        N/A |    454.760 

Note that there is some variability here. This second run with the same parameters (1 thread, 10 outstanding IOs) yielded slightly fewer IOPS. You can reduce this variability by running with longer duration or averaging multiple runs. More on that later.

With this system, we don’t seem to have a CPU bottleneck. The overall CPU utilization is around 2% and the busiest core is under 9% of usage. This system has 4 cores and anything with less than 25% (1/4) overall CPU utilization is probably not an issue. In other configurations, you might run into CPU core bottlenecks, though…

8. Tune queue depth for small random IOs, part 2

Now let’s perform the same tuning exercise for small random IOS with a system with better storage performance and less capable cores. For this one, we’ll continue to use the same PowerShell script. However, this is on system using an SSD for storage and 8 slower CPU cores. Here’s that same script again:

1..16 | % { 
   $param = "-o $_"
   $result = C:\DiskSpd\DiskSpd.exe -c1G -d10 -w0 -r -b8k $param -t1 -h -L C:\testfile.dat
   foreach ($line in $result) {if ($line -like "total:*") { $total=$line; break } }
   foreach ($line in $result) {if ($line -like "avg.*") { $avg=$line; break } }
   $mbps = $total.Split("|")[2].Trim()
   $iops = $total.Split("|")[3].Trim()
   $latency = $total.Split("|")[4].Trim()
   $cpu = $avg.Split("|")[1].Trim()  
   "Param $param, $iops iops, $mbps MB/sec, $latency ms, $cpu CPU"
}

Here’s the sample output from our second system:

Param -o 1, 7873.26 iops, 61.51 MB/sec, 0.126 ms, 3.96% CPU
Param -o 2, 14572.54 iops, 113.85 MB/sec, 0.128 ms, 7.25% CPU
Param -o 3, 23407.31 iops, 182.87 MB/sec, 0.128 ms, 6.76% CPU
Param -o 4, 31472.32 iops, 245.88 MB/sec, 0.127 ms, 19.02% CPU
Param -o 5, 32823.29 iops, 256.43 MB/sec, 0.152 ms, 20.02% CPU
Param -o 6, 33143.49 iops, 258.93 MB/sec, 0.181 ms, 20.71% CPU
Param -o 7, 33335.89 iops, 260.44 MB/sec, 0.210 ms, 20.13% CPU
Param -o 8, 33160.54 iops, 259.07 MB/sec, 0.241 ms, 21.28% CPU
Param -o 9, 36047.10 iops, 281.62 MB/sec, 0.249 ms, 20.86% CPU
Param -o 10, 33197.41 iops, 259.35 MB/sec, 0.301 ms, 20.49% CPU
Param -o 11, 35876.95 iops, 280.29 MB/sec, 0.306 ms, 22.36% CPU
Param -o 12, 32955.10 iops, 257.46 MB/sec, 0.361 ms, 20.41% CPU
Param -o 13, 33548.76 iops, 262.10 MB/sec, 0.367 ms, 20.92% CPU
Param -o 14, 34728.42 iops, 271.32 MB/sec, 0.400 ms, 24.65% CPU
Param -o 15, 32857.67 iops, 256.70 MB/sec, 0.456 ms, 22.07% CPU
Param -o 16, 33026.79 iops, 258.02 MB/sec, 0.484 ms, 21.51% CPU

As you can see, this SSD can deliver many more IOPS than the previous system which used multiple HDDs. We got consistently better performance as we increased the queue depth for the first few runs. As usual, after a certain number of outstanding IOs, adding more started giving us very little improvement until things flatten out completely and all we do is increase latency. This is coming from a single SSD. If you have multiple SSDs in Storage Spaces Pool or a RAID set, you might need to try even higher queue depths. Always make sure you increase –o parameter to reach the point where IOPS hit a peak and only latency increases.

So, in this setup, we seem to start losing steam at around 6 outstanding IOs and latency starts to ramp up more dramatically after queue depth reaches 8. Let’s see the full output for queue depth of 8 to get a good sense:

PS C:\> C:\DiskSpd\DiskSpd.exe -c1G -d10 -w0 -r -b8k -o8 -t1 -h -L C:\testfile.dat

Command Line: C:\DiskSpd\DiskSpd.exe -c1G -d10 -w0 -r -b8k -o8 -t1 -h -L C:\testfile.dat

Input parameters:

    timespan:   1
    -------------
    duration: 10s
    warm up time: 5s
    cool down time: 0s
    measuring latency
    random seed: 0
    path: 'C:\testfile.dat'
        think time: 0ms
        burst size: 0
        software and hardware cache disabled
        performing read test
        block size: 8192
        using random I/O (alignment: 8192)
        number of outstanding I/O operations: 8
        stride size: 8192
        thread stride size: 0
        threads per file: 1
        using I/O Completion Ports
        IO priority: normal

Results for timespan 1:
*******************************************************************************

actual test time:    10.00s
thread count:        1
proc count:        8

CPU |  Usage |  User  |  Kernel |  Idle
-------------------------------------------
   0|  99.06%|   2.97%|   96.09%|   0.94%
   1|   5.16%|   0.62%|    4.53%|  94.84%
   2|  14.53%|   2.81%|   11.72%|  85.47%
   3|  17.97%|   6.41%|   11.56%|  82.03%
   4|  24.06%|   5.16%|   18.91%|  75.94%
   5|   8.28%|   1.56%|    6.72%|  91.72%
   6|  16.09%|   3.91%|   12.19%|  83.90%
   7|   8.91%|   0.94%|    7.97%|  91.09%
-------------------------------------------
avg.|  24.26%|   3.05%|   21.21%|  75.74%

Total IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |      2928967680 |       357540 |     279.32 |   35753.26 |    0.223 |     0.051 | C:\testfile.dat (1024MB)
-----------------------------------------------------------------------------------------------------
total:        2928967680 |       357540 |     279.32 |   35753.26 |    0.223 |     0.051

Read IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |      2928967680 |       357540 |     279.32 |   35753.26 |    0.223 |     0.051 | C:\testfile.dat (1024MB)
-----------------------------------------------------------------------------------------------------
total:        2928967680 |       357540 |     279.32 |   35753.26 |    0.223 |     0.051

Write IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | C:\testfile.dat (1024MB)
-----------------------------------------------------------------------------------------------------
total:                 0 |            0 |       0.00 |       0.00 |    0.000 |       N/A

  %-ile |  Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
    min |      0.114 |        N/A |      0.114
   25th |      0.209 |        N/A |      0.209
   50th |      0.215 |        N/A |      0.215
   75th |      0.224 |        N/A |      0.224
   90th |      0.245 |        N/A |      0.245
   95th |      0.268 |        N/A |      0.268
   99th |      0.388 |        N/A |      0.388
3-nines |      0.509 |        N/A |      0.509
4-nines |      2.905 |        N/A |      2.905
5-nines |      3.017 |        N/A |      3.017
6-nines |      3.048 |        N/A |      3.048
7-nines |      3.048 |        N/A |      3.048
8-nines |      3.048 |        N/A |      3.048
    max |      3.048 |        N/A |      3.048

Again you note that there is some variability here. This second run with the same parameters (1 thread, 8 outstanding IOs) yielded a few more IOPS. We’ll later cover some tips on how to average out multiple runs.

You can also see that apparently one of the CPU cores is being hit harder than others. There is clearly a potential bottleneck. Let’s look into that…

9. Tune threads for small random IOs with CPU bottleneck

In this 8-core system, any overall utilization above 12.5% (1/8 of the total) means a potential core bottleneck when using a single thread. You can actually see in the CPU table in our last run that our core 0 is pegged at 99%. We should be able to do better with multiple threads. Let’s try increasing the number of threads with a matching reduction of queue depth so we end up with the same number of total outstanding IOs.

$o = 8
$t = 1
While ($o -ge 1) { 
   $paramo = "-o $o"
   $paramt = “-t $t”
   $result = C:\DiskSpd\DiskSpd.exe -c1G -d10 -w0 -r -b8k $paramo $paramt -h -L C:\testfile.dat
   foreach ($line in $result) {if ($line -like "total:*") { $total=$line; break } }
   foreach ($line in $result) {if ($line -like "avg.*") { $avg=$line; break } }
   $mbps = $total.Split("|")[2].Trim()
   $iops = $total.Split("|")[3].Trim()
   $latency = $total.Split("|")[4].Trim()
   $cpu = $avg.Split("|")[1].Trim()
   “Param $paramo $paramt, $iops iops, $mbps MB/sec, $latency ms, $cpu CPU"
   $o = $o / 2
   $t = $t * 2
}

Here’s the output:

Param -o 8 -t 1, 35558.31 iops, 277.80 MB/sec, 0.225 ms, 22.36% CPU
Param -o 4 -t 2, 37069.15 iops, 289.60 MB/sec, 0.215 ms, 25.23% CPU
Param -o 2 -t 4, 34592.04 iops, 270.25 MB/sec, 0.231 ms, 27.99% CPU
Param -o 1 -t 8, 34621.47 iops, 270.48 MB/sec, 0.230 ms, 26.76% CPU

As you can see, in my system, adding a second thread improved things a bit, reaching our best yet 37,000 IOPS without much of a change in latency. It seems like we were a bit limited by the performance of a single core. We call that being “core bound”. See below the full output for the run with two threads:

PS C:\> C:\DiskSpd\DiskSpd.exe -c1G -d10 -w0 -r -b8k -o4 -t2 -h -L C:\testfile.dat

Command Line: C:\DiskSpd\DiskSpd.exe -c1G -d10 -w0 -r -b8k -o4 -t2 -h -L C:\testfile.dat

Input parameters:

    timespan:   1
    -------------
    duration: 10s
    warm up time: 5s
    cool down time: 0s
    measuring latency
    random seed: 0
    path: 'C:\testfile.dat'
        think time: 0ms
        burst size: 0
        software and hardware cache disabled
        performing read test
        block size: 8192
        using random I/O (alignment: 8192)
        number of outstanding I/O operations: 4
        stride size: 8192
        thread stride size: 0
        threads per file: 2
        using I/O Completion Ports
        IO priority: normal

Results for timespan 1:
*******************************************************************************

actual test time:    10.00s
thread count:        2
proc count:        8

CPU |  Usage |  User  |  Kernel |  Idle
-------------------------------------------
   0|  62.19%|   1.87%|   60.31%|  37.81%
   1|  62.34%|   1.87%|   60.47%|  37.66%
   2|  11.41%|   0.78%|   10.62%|  88.75%
   3|  26.25%|   0.00%|   26.25%|  73.75%
   4|   8.59%|   0.47%|    8.12%|  91.56%
   5|  16.25%|   0.00%|   16.25%|  83.75%
   6|   7.50%|   0.47%|    7.03%|  92.50%
   7|   3.28%|   0.47%|    2.81%|  96.72%
-------------------------------------------
avg.|  24.73%|   0.74%|   23.98%|  75.31%

Total IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |      1519640576 |       185503 |     144.92 |   18549.78 |    0.215 |     0.419 | C:\testfile.dat (1024MB)
     1 |      1520156672 |       185566 |     144.97 |   18556.08 |    0.215 |     0.404 | C:\testfile.dat (1024MB)
-----------------------------------------------------------------------------------------------------
total:        3039797248 |       371069 |     289.89 |   37105.87 |    0.215 |     0.411

Read IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |      1519640576 |       185503 |     144.92 |   18549.78 |    0.215 |     0.419 | C:\testfile.dat (1024MB)
     1 |      1520156672 |       185566 |     144.97 |   18556.08 |    0.215 |     0.404 | C:\testfile.dat (1024MB)
-----------------------------------------------------------------------------------------------------
total:        3039797248 |       371069 |     289.89 |   37105.87 |    0.215 |     0.411

Write IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | C:\testfile.dat (1024MB)
     1 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | C:\testfile.dat (1024MB)
-----------------------------------------------------------------------------------------------------
total:                 0 |            0 |       0.00 |       0.00 |    0.000 |       N/A

  %-ile |  Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
    min |      0.088 |        N/A |      0.088
   25th |      0.208 |        N/A |      0.208
   50th |      0.210 |        N/A |      0.210
   75th |      0.213 |        N/A |      0.213
   90th |      0.219 |        N/A |      0.219
   95th |      0.231 |        N/A |      0.231
   99th |      0.359 |        N/A |      0.359
3-nines |      0.511 |        N/A |      0.511
4-nines |      1.731 |        N/A |      1.731
5-nines |     80.959 |        N/A |     80.959
6-nines |     90.252 |        N/A |     90.252
7-nines |     90.252 |        N/A |     90.252
8-nines |     90.252 |        N/A |     90.252
    max |     90.252 |        N/A |     90.252

You can see now that cores 0 and 1 are being used, with both at around 62% utilization. So we have effectively eliminated the core bottleneck that we had before.

For systems with more capable storage, it’s easier to get “core bound” and adding more threads can make a much more significant difference. As I mentioned, it’s important to keep an eye on the per-core CPU utilization (not only the total CPU utilization) to look out for these bottlenecks.

10. Multiple runs are better than one

One thing you might have notice with DiskSpd (or any other tools like it) is that the results are not always the same given the same parameters. Each run is a little different. For instance, let’s try running our “-b8K –o4 -t2” with the very same parameters a few times to see what happens:

1..8 | % { 
   $result = C:\DiskSpd\DiskSpd.exe -c1G -d10 -w0 -r -b8k -o4 -t2 -h -L C:\testfile.dat
   foreach ($line in $result) {if ($line -like "total:*") { $total=$line; break } }
   foreach ($line in $result) {if ($line -like "avg.*") { $avg=$line; break } }
   $mbps = $total.Split("|")[2].Trim()
   $iops = $total.Split("|")[3].Trim()
   $latency = $total.Split("|")[4].Trim()
   $cpu = $avg.Split("|")[1].Trim()
   “Run $_, $iops iops, $mbps MB/sec, $latency ms, $cpu CPU"
}

Here are the results:

Run 1, 34371.97 iops, 268.53 MB/sec, 0.232 ms, 24.53% CPU
Run 2, 37138.29 iops, 290.14 MB/sec, 0.215 ms, 26.72% CPU
Run 3, 36920.81 iops, 288.44 MB/sec, 0.216 ms, 26.66% CPU
Run 4, 34538.00 iops, 269.83 MB/sec, 0.231 ms, 36.85% CPU
Run 5, 34406.91 iops, 268.80 MB/sec, 0.232 ms, 37.09% CPU
Run 6, 34393.72 iops, 268.70 MB/sec, 0.214 ms, 33.71% CPU
Run 7, 34451.48 iops, 269.15 MB/sec, 0.232 ms, 25.74% CPU
Run 8, 36964.47 iops, 288.78 MB/sec, 0.216 ms, 30.21% CPU

The results have a good amount of variability. You can look at the standard deviations by specifying the -D option to check how stable things are. But, in the end, how can you tell which measurements are the most accurate? Ideally, once you settle on a specific set of parameters, you should run DiskSpd a few times and average out the results. Here’s a sample PowerShell script to do it, using the last set of parameters we used for the 8KB IOs:

$tiops=0
$tmbps=0
$tlatency=0
$tcpu=0
$truns=10
1..$truns | % {
   $result = C:\DiskSpd\DiskSpd.exe -c1G -d10 -w0 -r -b8k -o4 -t2 -h -L C:\testfile.dat
   foreach ($line in $result) {if ($line -like "total:*") { $total=$line; break } }
   foreach ($line in $result) {if ($line -like "avg.*") { $avg=$line; break } }
   $mbps = $total.Split("|")[2].Trim()
   $iops = $total.Split("|")[3].Trim()
   $latency = $total.Split("|")[4].Trim()
   $cpu = $avg.Split("|")[1].Trim()
   “Run $_, $iops iops, $mbps MB/sec, $latency ms, $cpu CPU"
   $tiops += $iops
   $tmbps += $mbps
   $tlatency += $latency
   $tcpu  += $cpu.Replace("%","")
}
$aiops = $tiops / $truns
$ambps = $tmbps / $truns
$alatency = $tlatency / $truns
$acpu = $tcpu / $truns
“Average, $aiops iops, $ambps MB/sec, $alatency ms, $acpu % CPU"

The script essentially runs DiskSpd 10 times, totaling the numbers for IOPs, throughput, latency and CPU usage, so it can show an average at the end. The $truns variable represents the total number of runs desired. Variables starting with $t hold the totals. Variables starting with $a hold averages. Here’s a sample output:

Run 1, 37118.31 iops, 289.99 MB/sec, 0.215 ms, 35.78% CPU
Run 2, 34311.40 iops, 268.06 MB/sec, 0.232 ms, 38.67% CPU
Run 3, 36997.76 iops, 289.04 MB/sec, 0.215 ms, 38.90% CPU
Run 4, 34463.16 iops, 269.24 MB/sec, 0.232 ms, 24.16% CPU
Run 5, 37066.41 iops, 289.58 MB/sec, 0.215 ms, 25.14% CPU
Run 6, 37134.21 iops, 290.11 MB/sec, 0.215 ms, 26.02% CPU
Run 7, 34430.21 iops, 268.99 MB/sec, 0.232 ms, 23.61% CPU
Run 8, 35924.20 iops, 280.66 MB/sec, 0.222 ms, 25.21% CPU
Run 9, 33387.45 iops, 260.84 MB/sec, 0.239 ms, 21.64% CPU
Run 10, 36789.85 iops, 287.42 MB/sec, 0.217 ms, 25.86% CPU
Average, 35762.296 iops, 279.393 MB/sec, 0.2234 ms, 28.499 % CPU

As you can see, it’s a good idea to capture multiple runs. You might also want to run each iteration for a longer time, like 60 seconds instead of just 10 second.

Using 10 runs of 60 seconds (10 minutes total) might seem a little excessive, but that was the minimum recommended by one of our storage performance engineers. The problem with shorter runs is that they often don’t give the IO subsystem time to stabilize. This is particularly true when testing virtual file systems (such as those in cloud storage or virtual machines) when files are allocated dynamically. Also, SSDs exhibit write degradation and can sometimes take hours to reach a steady state (depending on how full the SSD is). So it's a good idea to run the test for a few hours in these configurations on a brand new system, since this could drop your initial IOPs number by 30% or more.

11. DiskSpd and SMB file shares

You can use DiskSpd to get the same type of performance information for SMB file shares. All you have to do is run DiskSpd from an SMB client with access to a file share.

It is as simple as mapping the file share to a drive letter using the old “NET USE” command or the new PowerShell cmdlet “New-SmbMapping”. You can also use a UNC path directly in the command line, instead of using drive letters.

Here are an example using the HDD-based system we used as our first few examples, now running remotely:

PS C:\diskspd> C:\DiskSpd\DiskSpd.exe -c1000G -d10 -w0 -r -b8k -o10 -t1 -h -L \\jose1011-st1\Share1\testfile.dat

Command Line: C:\DiskSpd\DiskSpd.exe -c1000G -d10 -w0 -r -b8k -o10 -t1 -h -L \\jose1011-st1\Share1\testfile.dat

Input parameters:

        timespan:   1
        -------------
        duration: 10s
        warm up time: 5s
        cool down time: 0s
        measuring latency
        random seed: 0
        path: '\\jose1011-st1\Share1\testfile.dat'
                think time: 0ms
                burst size: 0
                software and hardware cache disabled
                performing read test
                block size: 8192
                using random I/O (alignment: 8192)
                number of outstanding I/O operations: 10
                stride size: 8192
                thread stride size: 0
                threads per file: 1
                using I/O Completion Ports
                IO priority: normal

Results for timespan 1:
*******************************************************************************

actual test time:       10.01s
thread count:           1
proc count:             4

CPU |  Usage |  User  |  Kernel |  Idle
-------------------------------------------
   0|  12.96%|   0.62%|   12.34%|  86.98%
   1|   0.00%|   0.00%|    0.00%|  99.94%
   2|   0.00%|   0.00%|    0.00%|  99.94%
   3|   0.00%|   0.00%|    0.00%|  99.94%
-------------------------------------------
avg.|   3.24%|   0.16%|    3.08%|  96.70%

Total IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |       158466048 |        19344 |      15.10 |    1933.25 |    5.170 |     6.145 | \\jose1011-st1\Share1\testfile.dat (1000GB)
-----------------------------------------------------------------------------------------------------
total:         158466048 |        19344 |      15.10 |    1933.25 |    5.170 |     6.145

Read IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |       158466048 |        19344 |      15.10 |    1933.25 |    5.170 |     6.145 | \\jose1011-st1\Share1\testfile.dat (1000GB)
-----------------------------------------------------------------------------------------------------
total:         158466048 |        19344 |      15.10 |    1933.25 |    5.170 |     6.145

Write IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  AvgLat  | LatStdDev |  file
-----------------------------------------------------------------------------------------------------
     0 |               0 |            0 |       0.00 |       0.00 |    0.000 |       N/A | \\jose1011-st1\Share1\testfile.dat (1000GB)
-----------------------------------------------------------------------------------------------------
total:                 0 |            0 |       0.00 |       0.00 |    0.000 |       N/A

  %-ile |  Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
    min |      3.860 |        N/A |      3.860
   25th |      4.385 |        N/A |      4.385
   50th |      4.646 |        N/A |      4.646
   75th |      5.052 |        N/A |      5.052
   90th |      5.640 |        N/A |      5.640
   95th |      6.243 |        N/A |      6.243
   99th |     12.413 |        N/A |     12.413
3-nines |     63.972 |        N/A |     63.972
4-nines |    356.710 |        N/A |    356.710
5-nines |    436.406 |        N/A |    436.406
6-nines |    436.406 |        N/A |    436.406
7-nines |    436.406 |        N/A |    436.406
8-nines |    436.406 |        N/A |    436.406
    max |    436.406 |        N/A |    436.406

This is an HDD-based storage system, so most of the latency comes from the local disk, not the remote SMB access. In fact, we achieved numbers similar to what we had locally before.

12. Conclusion

I hope you have learned how to use DiskSpd to perform some storage testing of your own. I encourage you to use it to look at the performance of the storage features in Windows Server 2012, Windows Server 2012 R2 and Windows Server Technical Preview. That includes Storage Spaces, SMB3 shares, Scale-Out File Server, Storage Replica and Storage QoS. Let me know if you were able to try it out and feel free to share some of your experiments via blog comments.

Thanks to Bartosz Nyczkowski, Dan Lovinger, David Berg and Scott Lee for their contributions to this blog post.

Storage Spaces Tiering

Storage Spaces has an interesting new feature introduced in Windows Server 2012 R2: you can create a single space using different types of disk (typically HDDs and SSDs) and it will automatically move hot data to the fast tier and cold data to the slow tier. You can read more about it at http://technet.microsoft.com/en-us/library/dn789160.aspx. You can also try my step-by-step instructions at http://blogs.technet.com/b/josebda/archive/2013/08/28/step-by-step-for-storage-spaces-tiering-in-windows-server-2012-r2.aspx.

Pinning Files to a Tier

While Storage Spaces will automatically track “heat” and move data to correct tier, there are situations where you already know ahead of time that a certain file will work best when placed in the fast tier (like when you have the base OS VHDX file for many differencing disks in a VDI setup) or the slow tier (like when you create a VHDX file specifically for backups). For those situations, you have the option to pin a file to a specific tier. That means Storage Spaces will place those pinned files on the tier you specified, tracking heat and moving automatically only the other files in the space.

Here are the cmdlets used to pin, unpin and report on pinned files with some sample parameters:

• Set-FileStorageTier -CimSession <computer> -FilePath <String> -DesiredStorageTierFriendlyName <String>
• Clear-FileStorageTier -CimSession <computer> -FilePath <String>
• Get-FileStorageTier -CimSession <computer> -Volume <CimInstance>

Tiers on a Scale-Out File Server

The cmdlets above are fine if you are using the Storage Spaces on a standalone system, but when you are using them with a Scale-Out File Server, they are not so convenient.

First of all, when running from a remote computer, they require you to know which server owns the storage space at that time. When you have your multiple spaces running on a Scale-Out File Server, they are naturally spread across the nodes of the cluster (that’s why we call it “Scale-Out”). You can, of course, query the cluster to find out which node owns the space, but that’s an extra step.

Another issue is that the cmdlets above require you to know the local path to the file you want to pin (or the volume you want to report on). When you’re using the Scale-Out File Server, you typically refer to files using the UNC path (also called the remote file path). For instance, you might refer a VHDX file as \\Server1\Share2\Folder3\File4.VHDX, even though the local path to that file on the cluster shared volume might be C:\ClusterStorage\Volume1\Shares\Share2\Folder3\File4.VHDX. There are ways to query the path behind a share and calculate the local path, but that’s again an extra step.

The third item is that the cmdlet requires you to know the name of the tier to pin the file. Each tiered storage space has two tiers (typically one associated with HDDs and the other associated with SSDs) and they each are independent objects. If you have 5 tiered storage spaces, you effectively have 10 tiers (2 per space), each with their own unique ID and friendly name. If you name things consistently, you can probably create a scheme where the name of the tier uses the name of the space plus the media type (SSD or HDD), but sometimes people get creative. So you need to gather that third piece of information before you can pin the file.

The last thing to keep in mind is that the pinning (or unpinning) will only happen after the Storage Spaces Tiering Optimization task is run. So, if you want this to happen immediately, you need to run that schedule task manually. It’s a simple command, but you should always remember to do it.

The SmbStorageTier.ps1 PowerShell Script

Faced with these issues, I decided to write a little PowerShell script that makes life easier for the Hyper-V or Scale-Out File Server administrator. Using the Cluster, SMB and Storage PowerShell cmdlets, the script helps you pin/unpin files to a storage tier in a Scale-Out File Server. To pin a file, all you need to specify is the UNC path and the media type. To unpin, just specify the UNC path. You can also get a report for files pinned on a specific file share (actually, on the volume behind that share).

Here are the script parameters used to pin, unpin and report on pinned files with some sample parameters:

• .\SmbStorageTier.ps1 –PinFile <String> –MediaType <hdd/ssd>
• .\SmbStorageTier.ps1 –UnpinFile <String>
• .\SmbStorageTier.ps1 –ReportShare <String>

The file path specified is the UNC path, starting with \\ and the name of the server. The same format that you use in Hyper-V when creating VMs. The media type is either HDD or SSD.

The script will basically take those parameters and calculate the information required to run the Set-FileStorageTier, Clear-FileStorageTier or Get-FileStorageTier. It will also kick off the Storage Spaces Tiering Optimization task, to make the change effective immediately. It’s not all that complicated to do using PowerShell, but it has the potential to save you some time if you pin/unpin files on a regular basis.

This script has been tested with both Windows Server 2012 R2 and the Windows Server Technical Preview.

Sample Output

To help you understand how this works, here are some examples of the script in action.

Note that it actually shows the input parameters, the calculated parameters required by the native Storage cmdlets and the command line it actually executes.

PS C:\> .\SmbStorageTier.ps1 -PinFile \\josebda-fs\share2\VM2\VM2.vhdx -MediaType ssd

Input parameters
File to pin: \\JOSEBDA-FS\SHARE2\VM2\VM2.VHDX
Media type: SSD

Calculated parameters
Local file path: C:\ClusterStorage\Volume2\Share2\VM2\VM2.VHDX
Node owning the volume: JOSEBDA-A3
Tier Name: Space2_SSDTier

Executing command:
Set-FileStorageTier -CimSession JOSEBDA-A3 -FilePath C:\ClusterStorage\Volume2\Share2\VM2\VM2.VHDX –DesiredStorageTierFriendlyName Space2_SSDTier

PS C:\> .\SmbStorageTier.ps1 -ReportShare \\josebda-fs\share2\

Input parameter
Share Name: \\JOSEBDA-FS\SHARE2\

Calculated parameters
Volume behind the share: \\?\Volume{37bb3bf3-80fc-4a43-bc79-37246e5d2666}\
Node owning the volume: JOSEBDA-A3

Executing command:
Get-FileStorageTier –CimSession JOSEBDA-A3 –VolumePath \\?\Volume{37bb3bf3-80fc-4a43-bc79-37246e5d2666}\ | Select *

PlacementStatus              : Partially on tier
State                        : Pending
DesiredStorageTierName       : Space2_SSDTier
FilePath                     : C:\ClusterStorage\Volume2\Share2\VM2\VM2.vhdx
FileSize                     : 9063890944
FileSizeOnDesiredStorageTier : 7987003392
PSComputerName               : JOSEBDA-A3
CimClass                     : ROOT/Microsoft/Windows/Storage:MSFT_FileStorageTier
CimInstanceProperties        : {DesiredStorageTierName, FilePath, FileSize, FileSizeOnDesiredStorageTier...}
CimSystemProperties          : Microsoft.Management.Infrastructure.CimSystemProperties

PS C:\> .\SmbStorageTier.ps1 -UnpinFile \\josebda-fs\share2\VM2\VM2.vhdx

Input parameter
File to unpin: \\JOSEBDA-FS\SHARE2\VM2\VM2.VHDX

Calculated parameters
Local Path: C:\ClusterStorage\Volume2\Share2\VM2\VM2.VHDX
Node owning the volume: JOSEBDA-A3

Executing command:
Clear-FileStorageTier -CimSession JOSEBDA-A3 -FilePath C:\ClusterStorage\Volume2\Share2\VM2\VM2.VHDX

PS C:\>

The Link to Download

Well, now I guess all you need is the link to download the script in the TechNet Script Center. Here it is: https://gallery.technet.microsoft.com/scriptcenter/SmbStorageTierps1-A-simple-934c0f22

Let me know what you think. Add your comments to the blog or use the Q&A section in the TechNet Script Center link above.

With the release of Windows Server Technical Preview, a common question comes to the surface again. What kind of lab hardware do I need to play with it?

Ideally, you would get one of those new Cloud Platform System (CPS) racks with a 32-node Hyper-V cluster, a 4-node Scale-Out File Server and 4 JBODs with 60 disks each. However, not everyone can afford a lab like that :-).

I am recycling some of my previous step-by-step guides and offer some basic options on how to test the Windows Server 2012 R2 and Windows Server Technical Preview features.

IMPORTANT NOTE: Some of these configurations are not supported and are not recommended for production deployments. Please see each linked blog post for details.

Option 1 – Virtual Machines

You can do a lot with just a single physical machine running Hyper-V. With enough RAM and a nice Core i7 CPU you can configure half a dozen VMs and try out many scenarios.

This would be enough to to test Failover Clustering, Scale-Out File Server, Shared VHDX, Scale-Out Rebalancing and Storage Replica. You can even test some basic Storage Spaces and Storage QoS capabilities.

Obviously, you can’t do much testing of Hyper-V itself and certainly no Hyper-V clustering features. This will also not showcase storage performance at all, since you’re limited by hardware.

This is the setup I used many times on my beefy laptop with 32GB of RAM and used for many demos in the past.

Details at http://blogs.technet.com/b/josebda/archive/2013/07/31/windows-server-2012-r2-storage-step-by-step-with-storage-spaces-smb-scale-out-and-shared-vhdx-virtual.aspx

Option 2 – Physical Machines, basic networking

If you’re testing new Hyper-V features related to clustering, you can’t run virtualized. This would require at least two physical Hyper-V hosts.

Also, might want to test how Storage Spaces works with a physical JBOD plus a mix of HDD/SSD to test storage performance, tiering, etc.

A setup like this will be ideal to test Hyper-V high availability, Storage Spaces performance and the full capabilities of the new Storage QoS.

To do some basic testing you don't need high-end hardware. I've done a lot of testing with basic (even desktop-class) hardware. You do want to make they properly support Hyper-V.

Details at http://blogs.technet.com/b/josebda/archive/2013/07/31/windows-server-2012-r2-storage-step-by-step-with-storage-spaces-smb-scale-out-and-shared-vhdx-physical.aspx

While the setup in the link above suggests the use of RDMA, this will work fine with regular networking, even 1GbE. It will obviously perform accordingly.

Option 3 – Physical Machines, RDMA networking, all SSD

You can go one step further if you add RDMA networking to you setup. This means you need some RDMA cards and a more expensive switch. I might be a little noisier as well.

You might even go as far as to have multiple RDMA NICs so you can try out SMB Multichannel at high speeds. At this point you might want to move to an all-SSD configuration.

At that point you want to make sure the hardware for both the Hyper-V host and the File Server are server-class equipment with good performance, fast CPUs, lots of RAM. 

Details at http://blogs.technet.com/b/josebda/archive/2014/03/10/smb-direct-and-rdma-performance-demo-from-teched-includes-summary-powershell-scripts-and-links.aspx

This is probably the simplest configuration to showcase extremely high performance. Note that the example above does not include failover clustering.

Option 4 – CPS

Beyond that, you are now in CPS territory, which includes multiple Hyper-V hosts, multiple file servers, dual RDMA paths, multiple JBODs, tiering, the whole thing.

Details at http://www.microsoft.com/cps

Bonus – Option 5 – Azure VMs

An interesting option is to skip the physical lab completely and go all Azure with your testing lab.

It's fairly easy to configure an Azure VM as a standalone file server and use Azure data disks to experiment with Storage Spaces. You can also try out Storage Replica.

With a little care on how you configure your Azure networking, you can also setup a Scale-Out File Server using iSCSI for your shared storage.

Details at http://blogs.technet.com/b/josebda/archive/2014/03/29/deploying-a-windows-server-2012-r2-scale-out-file-server-cluster-using-azure-vms.aspx

I hope that gives you a few ideas on how to get started with your testing.

Introduction

If you have SMB File Servers running Windows Server 2003, you are probably already aware that the extended support for that OS will end on July 14, 2015. You can read more about this at: http://blogs.technet.com/b/server-cloud/archive/2014/08/26/best-practices-for-windows-server-2003-end-of-support-migration.aspx.

If you're still using Windows Server 2003, you should be planning your migration to a newer version right now. The easiest path to migrate an old SMB File Server would be to use a virtual machine to replace your old box and move the data to the new VM. While it is fairly straightforward to perform the migration, you do have to be careful about it, since it does mean moving data around and requires at least a little bit of down time.

I’m glad you’re reading this, since it indicates you’re taking the steps to retire your old server before it falls out of support.

The Steps

To test this migration scenario, I configured a 32-bit Windows Server 2003 machine (which I called FILES) and a Hyper-V virtual machine running Windows Server 2012 R2. I also configured a domain controller running Windows Server 2012 R2 and joined both machines to a single domain.

In general, here are the steps I used to test and capture the details on how to perform the migration:

1. Configure accounts/groups in Active Directory *
2. Configure your WS2003 box *
3. Prepare for migration (initial data copy prior to final migration)
4. Export the share information
5. Changes happen after Step 3 *
6. Rename Windows Server 2003
7. Final data migration pass
8. Create shares at the destination
9. Rename the WS2012 R2
10. Verify you are all done

Items marked with * are needed only for simulation purposes and should not be executed in your existing environment already running a Windows Server 2003 File Server.

Find below the details for each of the steps above. Note that I tried to capture the commands I used in my environment, but you will obviously need to adjust the server names and paths as required in your specific configuration.

Step 1 – Configure accounts/groups in Active Directory

This step creates the several users and groups for the domain that we’ll use in the script.

This step should be run from an elevated PowerShell prompt on the test domain controller running Windows Server 2012 R2.

Older Windows Server versions for the DC are fine, but I cannot vouch the PowerShell below working on all older versions.

IMPORTANT: These commands should only be used for a test environment simulation. Do not run on your production environment.

$cred = get-credential
1..99 | % { New-ADUser -Name User$_ -AccountPassword $cred.password -CannotChangePassword $true -DisplayName "Test $_" -Enabled $true -SamAccountName User$_ }
1..99 | % { New-ADGroup -DisplayName "Project $_" -Name Project$_ -GroupCategory Security -GroupScope Global }
1..99 | % { $Group = $_; 1..99 | % { Get-ADGroup Project$Group | Add-ADGroupMember -Members User$_ } }

Step 2 – Configure your WS2003 box

This step creates several folders and shares, with different permissions at the share and the file system level. This simulates a production environment and helps test that files, folders, shares and permissions are being properly migrated.

This step should be run from a command prompt on the test Windows Server 2003 File Server. In the script, JOSE is the name of the domain.

IMPORTANT: These commands should only be used for a test environment simulation. Do not run on your production environment.

md C:\homefolder
for /L %%a in (1,1,99) do md C:\homefolder\user%%a
for /L %%a in (1,1,99) do NET SHARE share%%a=C:\homefolder\user%%a /GRANT:JOSE\Administrator,FULL /GRANT:JOSE\user%%a,FULL
for /L %%a in (1,1,99) do echo y | cacls C:\homefolder\user%%a /E /G JOSE\Administrator:F
for /L %%a in (1,1,99) do echo y | cacls C:\homefolder\user%%a /E /G JOSE\user%%a:F

md c:\projects
for /L %%a in (1,1,99) do md C:\projects\project%%a
for /L %%a in (1,1,99) do NET SHARE project%%a=C:\projects\project%%a /GRANT:JOSE\Administrator,FULL /GRANT:JOSE\Project%%a,FULL
for /L %%a in (1,1,99) do echo y | cacls c:\projects\project%%a /E /G JOSE\Administrator:F
for /L %%a in (1,1,99) do echo y | cacls c:\projects\project%%a /E /G JOSE\project%%a:F

for /L %%a in (1,1,99) do xcopy c:\windows\media\*.mid C:\homefolder\user%%a
for /L %%a in (1,1,99) do xcopy c:\windows\media\*.mid c:\projects\project%%a

Step 3 – Prepare for migration

This step performs an initial data copy from the Windows Server 2003 File Server to the Windows Server 2012 R2 machine prior to the final migration.

By doing this initial copy with the old file server still accessible to users, you minimize the downtime required for the final copy. If there are issues with open files or other errors during this step, that is OK. You will have a chance to grab those files later.

You should make sure to include all the folders used for all your file shares. In this example I am assuming relevant files are in the folders called c:\homefolder and c:\projects.

IMPORTANT: You must use the same drive letters and the exact same paths on your new Windows Server 2012 R2 server. If you don't, the share information won't match and your migration will not work.

IMPORTANT: This migration process only works if you only use domain accounts and domain groups for your permissions. If you are using local accounts for the file share or file system permissions, the permissions will not be migrated by ROBOCOPY.

In case you’re not familiar with ROCOBOPY, here are the details about the parameters used:

      /e – Copy subdirectories, including empty ones
     /xj – Exclude junction points
    /r:2 – 2 retries
    /w:5 – 5 second wait between retries
      /v – Verbose output for skipped files
     /it – Include tweaked files (identical size/timestamp, but different attributes)
  /purge – Delete destination files/directories that no longer exist in source
/copyall – Copy data, attributes, timestamps, security (ACLs), owner, auditing info

Run this step at the Windows Server 2012 R2 server from an elevated command prompt.

md C:\homefolder
ROBOCOPY /e /xj /r:2 /w:5 /v /it /purge /copyall \\FILES\c$\homefolder c:\homefolder

md c:\projects
ROBOCOPY /e /xj /r:2 /w:5 /v /it /purge /copyall \\FILES\c$\projects c:\projects

Step 4 – Export the share information

This step exports the share information from the registry of the Windows Server 2003 machine. This will include share names, share path and share security (ACLs). There are more details on this export procedure at http://support.microsoft.com/kb/125996.

This command should be run from a command prompt on the test Windows Server 2003 File Server.

IMPORTANT: This migration process only works if you only use domain accounts and domain groups for your permissions. If you are using local accounts for the file share or file system permissions, the permissions will not be migrated by this registry export.

reg export HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Shares c:\export.reg

 

Step 5 – Changes happen after Step 3 (simulating an existing environment) (run from WS2003 file server)

This step simulates changes being applied to the files after the initial copy in step 3. Since some time will pass between steps 3 and 6, we expect that users will still be making changes to their files and adding new files. This simulated step makes sure the command are able to property capture those changes.

This step should be run from a command prompt on the test Windows Server 2003 File Server.

IMPORTANT: These commands should only be used for a test environment simulation. Do not run on your production environment.

for /L %%a in (1,1,99) do xcopy c:\windows\media\r*.wav c:\homefolder\user%%a
for /L %%a in (1,1,99) do xcopy c:\windows\media\r*.wav c:\projects\project%%a

 

Step 6 – Rename Windows Server 2003 (***DOWNTIME STARTS HERE***)

This step asks you to rename the Windows Server 2003 computer and reboot it. This will mark the beginning of downtime for your File service.

Since Windows Server 2003 did not ship with a command-line option to perform this operation, use the GUI to manually rename the machine from FILES to XFILES. This assumes that FILES is the name of the existing file server (users access data using \\FILES\<sharename>) and XFILES is an unused name in your network. At this point, your FILES file server will become unavailable.

If you want to automate this step, download the Support Tools from http://www.microsoft.com/en-us/download/details.aspx?id=15326 and use the command below from the Windows Server 2003 machine:

NETDOM RENAMECOMPUTER /NEWNAME XFILES /REBOOT /FORCE

Step 7 – Final data migration pass

This step copies the changes in the shares (changed files, new files) after the initial copy. Since this happens after the system is renamed and rebooted, there should be no more users in the system and there should be no further problems with files being in use during the copy.

We’re using the same parameters as before and ROBOCOPY will basically copy just what changed since the initial copy. If the initial copy was not too far back, you will have fewer changes and this step will be shorter.

IMPORTANT: Since this is the last copy, you should watch for any failures and repeat the copy until there are no issues with any files you care for.

Run this from the Windows Server 2012 R2 server from an elevated command prompt.

ROBOCOPY /e /xj /r:2 /w:5 /v /it /purge /copyall \\XFILES\c$\homefolder c:\homefolder
ROBOCOPY /e /xj /r:2 /w:5 /v /it /purge /copyall \\XFILES\c$\projects c:\projects

 

Step 8  – Create shares at the destination

This step imports the share configuration from the Windows Server 2003 system, using the file you created on step 4.

Run this from the Windows Server 2012 R2 server from an elevated command prompt.

reg import \\XFILES\c$\export.reg

 

Step 9 – Rename the WS2012 R2

In this step, the Windows Server 2012 R2 system is renamed to the same name as the old Windows Server 2003 system and the migration is done.

As soon as the system is rebooted, the clients will be able to access the file shares in the new system and the downtime ends.

Run this from the Windows Server 2012 R2 server from an elevated PowerShell prompt.

Rename-Computer -NewName FILES -Restart -Force

 

Step 10  – Verify you are all done! (***DOWNTIME ENDS HERE***)

At this point, you migration is complete and you’re basically using these commands to verify that the shares were properly migrated. The commands also sample the permissions in some of the shares and folders to verify that portion worked as well.

Run this from the Windows Server 2012 R2 server from an elevated PowerShell prompt.

Get-SmbShare
Get-SmbShareAccess Share23
Get-SmbShareAccess Project9
Get-Acl c:\homefolder\user23 | Format-List Path, AccessToString
Get-Acl c:\projects\project9 | Format-List Path, AccessToString

 

Conclusion

I highly recommend that you setup a test environment before trying this on your production Windows Server 2003 File Server. You test environment should also mimic your production environment as closely as possible. This way you can learn details about the procedure and customize the scripts to the details of your environment.

Good luck on your migration!

I was looking into the list of Azure PowerShell cmdlets and I could not find a page with all cmdlets and parameters in a single list. There are individual pages for each cmdlet on TechNet, but not all cmdlets in one page. There is a list of all cmdlets in one page, but that does not include the parameters as well.

However, since we have the Get-Command and Get-Help cmdlets, that is something that we can solve by writing a script. So, after installing the latest (as of 11/14/14) Azure PowerShell cmdlets, I wrote a little script to output all cmdlet names and parameters to HTML. Here’s that script:

$Cmdlet = "" | Select Name, Parameters
Get-command -Module Azure | % {
  Get-Help $_ | % {
    $ParameterSetCount = 1
    $CmdletName = $_.Name
    $ParameterSets = $_.syntax.syntaxItem
    $ParameterSets | % {

      $Cmdlet.Name=$CmdletName
      If ($ParameterSets.Count -gt 1) {
        $Cmdlet.Name+=” (“+$ParameterSetCount+”)”
        $ParameterSetCount++
      }

      $Parameters = $_.parameter
      $StringPar=””
      If ($Parameters) {
        $First=$true
        $Parameters | % {
          If ($First) {$First=$false} else {$StringPar+="%%"}
          $StringPar+= “ -“+$_.name+” “;
          if ($_.parameterValue -and ($_.parameterValue –notlike “Switch*”)) {
            $StringPar+= ”<”+$_.parameterValue+”> “
          }
        }
      }
      $StringPar=$StringPar.Replace(“Nullable``1”,””)
      $StringPar=$StringPar.Replace(“System.[[System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]”,”Int32”)
      $StringPar=$StringPar.Replace(“System.[[System.Boolean, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]”,”Boolean”)
      $Cmdlet.parameters=$StringPar;

      $Cmdlet 
    }
  }
} | ConvertTo-HTML | Out-File D:\AzureCmdlets.HTML

I ended up having to make a few adjustments to make things work better for my specific purpose:

• If a cmdlet has more than one form (more than one parameter set), I listed them with a number next to the cmdlet name
• I added the data type next to the parameter name, enclosed in <angle brackets> (except for switch parameters)
• Certain data types were showing some additional info that I did not need (like “Nullable”), which I removed
• The script adds a “%%” mark between parameters. I later replaced those with HTML line breaks (“<BR>”)

Here is the output of the script, with some minor adjustments to the HTML: