Storage Spaces Direct Step by Step: Part 3 Network Infrastructure

Deploying S2D on a 4-node cluster. QoS & Hyper-V virtual adaptors. Switch Enabled Teaming virtual switch & SMB Multipath infrastructure. Part 3 of 4.


This article is the third of a four part series.

  1. Core Cluster
  2. Troubleshooting Storage Clusters
  3. Configuring Storage Network Infrastructures [this article]
  4. Managing Storage Clusters

In Part 1 Core Cluster the base functions of Windows Storage Spaces Direct was described. A storage infrastructure is complex with multiple hardware and software systems. Once the base functionality has been tested completely, the steps here in Part 3 Network Infrastructure can be completed. Layering the configurations in this way will allow the administrator to better understand and troubleshoot the storage environment before adding additional technologies.

  • Configure QoS networking
  • Configure the Hyper-V virtual adapters
  • Create the Switch Enabled Teaming virtual switch
  • Create the Multipath infrastructure
Hyper-v RDMA and SET switch network architecture
Hyper-v RDMA and SET switch network architecture

The diagram shows the Hyper-V network components and the storage paths for VMs accessing storage.

  • Switch Embedded Teaming (SET) if an advanced virtual switch which supports teamed physical network ports as well as Direct, SMB Multi-Channel and RDMA technologies.
  • pNIC is RDMA enabled physical ports, for example hardware from (Mellanox, Chelsio, HP, Dell and Intel).
  • Storage network traffic uses the SMB protocol. Hyper-V and Storage Spaces work under-the-hood automatically routing RDMA storage access over internal Hyper-V RDNA enabled virtual adaptors. VMs access storage using the local path [C:StorageCluster.
  • VMs access storage on non-local storage nodes across the pNIC physical network LAN.

Configure Network for RDMA

Perform the following procedure separately on each node in your .

Install Feature Prerequisites

Perform this step on each node.


This is not required for servers with iWarp network adapters.

Install-WindowsFeature -Name Data-Center-Bridging -ComputerName HC-Node1

Configure QoS policy for SMB Direct

Perform all of the following steps 1-11 in this section on each node.

  1. Set up a policy to tag SMB-Direct packets with a priority tag.

In this example, we use priority tag “3”. Keep in mind that DCB's QoS policies apply globally, so SMB packets sent using RDMA (“NetDirect” is synonymous with RDMA) will always get tagged with the value “3” no matter what interface they are sent on.


While this guide uses the tag value 3, any tag value between 1 and 7 inclusive can be used as long as it is used everywhere through the network in both the hosts and the switches/routers.

New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3

The response should look like:

Name : SMB
Owner : Group Policy (Machine)
NetworkProfile : All
Precedence : 127
JobObject :
NetDirectPort : 445
PriorityValue : 3
  1. Enable Priority-based Flow Control (PFC) for the SMB-Direct traffic
Enable-NetQosFlowControl -priority 3

If the PFC enablement succeeds there is no response.

  1. Reserve some bandwidth for the SMB-Direct storage traffic

This example uses 50%, but you may want to reserve less or more depending on what you expect the ratio of storage verses other types of traffic.

New-NetQosTrafficClass "SMB" -priority 3 -bandwidthpercentage 50 -algorithm ETS

The response to the traffic class creation should look like:

Name Algorithm Bandwidth(%) Priority PolicySet IfIndex IfAlias
---- --------- ------------ -------- --------- ------- -------
SMB  ETS       50           3        Global
  1. Set the policies on each physical interface
Enable-NetAdapterQos -InterfaceAlias Provider1,Provider2
  1. Disable Data Center Bridging Exchange (DCBX) traffic

Refer to the Microsoft documentation Managing the Local DCBX Willing State.


This is not used by iWarp

Enable- Set-NetQosDcbxSestting Provider1,Provider2 -Willing $False
  1. Disable all non-SMB/RDMA traffic goes without a priority tag
New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 0
  1. Disable PFC traffic on the non-SMB traffic
Disable-NetQosFlowControl -priority 0,1,2,4,5,6,7
  1. Test NetQosFlowControl configuration

Which should return “True” for the priority tags for which you have turned on PFC and “False” for the rest, e.g.,

Priority Enabled PolicySet IfIndex IfAlias
-------- ------- --------- ------- -------
0        False   Global
1        False   Global
2        False   Global
3        True    Global
4        False   Global
5        False   Global
6        False   Global
7        False   Global
  1. Test the NetAdapterQos configuration
Get-NetAdapterQos -Name Provider1,Provider2

which returns:

Name : Provider1
Enabled : True
Capabilities :                                     Hardware Current
                                                   -------- -------
                             MacSecBypass        : NotSupported NotSupported
                             DcbxSupport         : None None
                             NumTCs(Max/ETS/PFC) : 8/8/8 8/8/8
OperationalTrafficClasses  : TC TSA Bandwidth Priorities
                             -- --- --------- ----------
                              0 ETS 50%       0-2,4-7
                              1 ETS 50%       3
OperationalFlowControl     : Priority 3 Enabled
OperationalClassifications : Protocol Port/Type Priority
                             -------- --------- --------
                             Default            0
                             NetDirect 445      3
Name                       : Provider2
Enabled                    : True
Capabilities               :                       Hardware     Current
                                                   --------     -------
                             MacSecBypass        : NotSupported NotSupported
                             DcbxSupport         : None         None
                             NumTCs(Max/ETS/PFC) : 8/8/8        8/8/8
OperationalTrafficClasses  : TC TSA Bandwidth Priorities
                             -- --- --------- ----------
                              0 ETS 50%       0-2,4-7
                              1 ETS 50%       3
OperationalFlowControl     : Priority 3 Enabled
OperationalClassifications : Protocol  Port/Type Priority
                             --------  --------- --------
                             Default   0
                             NetDirect 445 3
  1. Enable RDMA on the physical adapters
Enable-NetAdapterRdma Provider1,Provider2

The return should look like

Name InterfaceDescription        Enabled
---- --------------------        -------
NIC1 Chelsio Network Adapter     True
NIC2 Chelsio Network Adapter #2  True
  1. Confirm that SMB also sees these interfaces as RDMA-capable

The result should look like:

Interface Index RSS Capable RDMA Capable Speed   IpAddresses
--------------- ----------- ------------ -----   -----------
3               True        True         40 Gbps {fe80::e14f:b55:b3dc:b03c,}
7               True        True         40 Gbps {fe80::9ce6:c07:9aab:d0f4,}

If for some reason the RDMA Capable column in the Get-SmbClientNetworkInterface output shows False, it may require a reboot of the host to get SMB to update the value.

Create the SET vSwitch

  1. Create the SET switch on each node
New-VMSwitch -Name Provider -NetAdapterName Provider1,Provider2 -AllowManagementOS $true -EnableEmbeddedTeaming $true

The response should be something like:

Name     SwitchType NetAdapterInterfaceDescription
----     ---------- ------------------------------
Provider External   Teamed-Interface
  1. Add Hyper-V vNICs to the SET virtual switch

This configures a virtual (vNIC) attached to the virtual switch that you just configured for the management OS to use.


Because SET is integrated into the Hyper-V Virtual Switch, you cannot use SET Switch inside of a virtual machine.

Add-VMNetworkAdapter –SwitchName Provider –Name hvNIC –managementOS

The hvNIC will be the virtual available to Virtual Machines for client traffic.

Create the SMB-Multipath vNICs

Storage access for Hyper-V VMs will use RDMA using the following vNIC configurations.

  1. Create the Hyper-V vNICs. Two NICs are created to supply SMB Multipath two paths
Add-VMNetworkAdapter –SwitchName Provider –Name SMB1 –managementOS
Add-VMNetworkAdapter –SwitchName Provider –Name SMB2 –managementOS


Some network switches and iWarp may require VLANs. The following sets a VLAN.

Get-VMNetworkAdapter -ManagementOS SMB1 | Set-VMNetworkAdapterVLan -Access -VlanID 20
Get-VMNetworkAdapter -ManagementOS SMB2 | Set-VMNetworkAdapterVLan -Access -VlanID 20
  1. Enable the SMB Multipath vNICs for RDMA
Enable-NetAdapterRDMA -Name “vEthernet (SMB1)”,”vEthernet (SMB2)”


SMB1 and SMB2 should not be attached directly to Virtual Machines. When creating a storage device inside a , the access to Storage Spaces Direct volumes will use these RDMA devices. This is managed automatically by Hyper-V.

Next Steps

  • The hyper-converged environment is complete and workloads can be deployed.
  • Use the Manager GUI or PowerShell to deploy and manage virtual machines.
  • Create storage volumes, rename the new folder, and then use the direct volume folder path:

4 thoughts on “Storage Spaces Direct Step by Step: Part 3 Network Infrastructure”

  1. Hello
    I tested this artical with Mellanox ConnectX-3 IPoIB Adapter.

    “Step 4 Set the policies on each physical interface”

    PS: Enable-NetAdapterQos -InterfaceAlias Provider1

    response is

    Enable-NetAdapterQos : ‘Provider1 ‘과(와) 같은 ‘Name’ 속성을 가진 MSFT_NetAdapterQosSettingData 개체가 없습니다. 속성 값을 검증하고 다시 시도하십시오.
    위치 줄:1 문자:1
    + Enable-NetAdapterQos -InterfaceAlias “Provider1 ”
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : ObjectNotFound: (Provider1 :String) [Enable-NetAdapterQos], CimJobException
    + FullyQualifiedErrorId : CmdletizationQuery_NotFound_Name,Enable-NetAdapterQos

    I can’t find any solution for this error message.

    Help me PLZ

  2. I think this is an informative post and it is very useful and knowledgeable. therefore, I would like to thank you for the efforts you have made in writing this article.

Leave a Reply

Your email address will not be published. Required fields are marked *