Select Page

Configuring Storage Spaces Direct Step by Step: Part 2 Deployment

Introduction

This article is the second of a four part series.

  1. Core Cluster
  2. Deployment (this article)
  3. Management
  4. Troubleshooting

In Part 1 Core Cluster the base functions of Windows Storage Spaces Direct was described. A storage infrastructure is complex with multiple hardware and software systems. Once the base functionality has been tested completely, the steps here in Part 2 Deployment can be completed. Layering the configurations in this way will allow the administrator to better understand and troubleshoot the storage environment before adding additional technologies.

  • Configure QoS networking
  • Configure the Hyper-V virtual adapters
  • Create the Switch Enabled Teaming virtual switch
  • Create the SMB Multipath infrastructure

Hyper-v RDMA and SET switch network architecture

Configure the network for RDMA

Perform the following procedure separately on each node in your cluster.

Install Feature Prerequisites

Install on each node.

Note: Not required or desired for iWarp network adapters.
Install-WindowsFeature -Name Data-Center-Bridging -ComputerName HC-Node1

Configure QoS policy for SMB Direct

  1. Set up a policy to tag SMB-Direct packets with a priority tag.

In this example we use priority tag “3”. Keep in mind that DCB’s QoS policies apply globally, so SMB packets sent using RDMA (“NetDirect” is synonymous with RDMA) will always get tagged with the value “3” no matter what interface they are sent on.

Note: While this guide uses the tag value 3, any tag value between 1 and 7 inclusive can be used as long as it is used everywhere through the network in both the hosts and the switches/routers.
New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3

The response should look like:

Name : SMB
Owner : Group Policy (Machine)
NetworkProfile : All
Precedence : 127
JobObject :
NetDirectPort : 445
PriorityValue : 3
  1. Enable PFC for the SMB-Direct traffic
Enable-NetQosFlowControl -priority 3

If the PFC enablement succeeds there is no response.

  1. Reserve some bandwidth for the SMB-Direct storage traffic

This example uses 50%, but you may want to reserve less or more depending on what you expect the ratio of storage verses other types of traffic.

New-NetQosTrafficClass "SMB" -priority 3 -bandwidthpercentage 50 -algorithm ETS

The response to the traffic class creation should look like:

Name Algorithm Bandwidth(%) Priority PolicySet IfIndex IfAlias
---- --------- ------------ -------- --------- ------- -------
SMB  ETS       50           3        Global
  1. Set the policies on each physical interface
Enable-NetAdapterQos -InterfaceAlias Provider1,Provider2
  1. Disable DCBX traffic
Enable- Set-NetQosDcbxSestting Provider1,Provider2 -Willing $False
  1. Disable all non-SMB/RDMA traffic goes without a priority tag
New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 0
  1. Disable PFC traffic on the non-SMB traffic
Disable-NetQosFlowControl -priority 0,1,2,4,5,6,7
  1. Test NetQosFlowControl configuration
Get-NetQosFlowControl

Which should return “True” for the priority tags for which you have turned on PFC and “False” for the rest, e.g.,

Priority Enabled PolicySet IfIndex IfAlias
-------- ------- --------- ------- -------
0        False   Global
1        False   Global
2        False   Global
3        True    Global
4        False   Global
5        False   Global
6        False   Global
7        False   Global
  1. Test the NetAdapterQos configuration
Get-NetAdapterQos -Name Provider1,Provider2

which returns:

Name : Provider1
Enabled : True
Capabilities :                                     Hardware Current
                                                   -------- -------
                             MacSecBypass        : NotSupported NotSupported
                             DcbxSupport         : None None
                             NumTCs(Max/ETS/PFC) : 8/8/8 8/8/8
OperationalTrafficClasses  : TC TSA Bandwidth Priorities
                             -- --- --------- ----------
                              0 ETS 50%       0-2,4-7
                              1 ETS 50%       3
OperationalFlowControl     : Priority 3 Enabled
OperationalClassifications : Protocol Port/Type Priority
                             -------- --------- --------
                             Default            0
                             NetDirect 445      3
Name                       : Provider2
Enabled                    : True
Capabilities               :                       Hardware     Current
                                                   --------     -------
                             MacSecBypass        : NotSupported NotSupported
                             DcbxSupport         : None         None
                             NumTCs(Max/ETS/PFC) : 8/8/8        8/8/8
OperationalTrafficClasses  : TC TSA Bandwidth Priorities
                             -- --- --------- ----------
                              0 ETS 50%       0-2,4-7
                              1 ETS 50%       3
OperationalFlowControl     : Priority 3 Enabled
OperationalClassifications : Protocol  Port/Type Priority
                             --------  --------- --------
                             Default   0
                             NetDirect 445 3
  1. Enable RDMA on the physical adapters
Enable-NetAdapterRdma Provider1,Provider2

The return should look like

Name InterfaceDescription        Enabled
---- --------------------        -------
NIC1 Chelsio Network Adapter     True
NIC2 Chelsio Network Adapter #2  True
  1. Confirm that SMB also sees these interfaces as RDMA-capable
Get-SmbClientNetworkInterface

The result should look like:

Interface Index RSS Capable RDMA Capable Speed   IpAddresses
--------------- ----------- ------------ -----   -----------
3               True        True         40 Gbps {fe80::e14f:b55:b3dc:b03c, 192.168.1.3}
7               True        True         40 Gbps {fe80::9ce6:c07:9aab:d0f4, 192.168.2.3}

If for some reason the RDMA Capable column in the Get-SmbClientNetworkInterface output shows False, it may require a reboot of the host to get SMB to update the value.

Create the SET vSwitch

  1. Create the SET switch
New-VMSwitch -Name Provider -NetAdapterName Provider1,Provider2 -AllowManagementOS $true -EnableEmbeddedTeaming $true

The response should be something like:

Name     SwitchType NetAdapterInterfaceDescription
----     ---------- ------------------------------
Provider External   Teamed-Interface
  1. Create the virtual switch connected to both physical network adapters, and enable the Switch Embedded Teaming (SET).
New-VMSwitch –Name SETswitch –NetAdapterName Provider1, Provider2 -EnableEmbeddedTeaming $true
  1. Add Hyper-V vNICs to the virtual switch

This configures a virtual NIC (vNIC) from the virtual switch that you just configured for the management OS to use.

Because SET is integrated into the Hyper-V Virtual Switch, you cannot use SET Switch inside of a virtual machine (VM).

Add-VMNetworkAdapter –SwitchName SETswitch –Name hvNIC –managementOS

The hvNIC will be the virtual NIC available to Virtual Machines for client traffic.

Create the SMB-Multipath vNICs

Storage access for Hyper-V VMs will use RDMA using the following vNIC configurations.

  1. Create the Hyper-V vNICs. Two NICs are created to supply SMB Multipath two paths
Add-VMNetworkAdapter –SwitchName Provider –Name SMB1 –managementOS Add-VMNetworkAdapter –SwitchName Provider –Name SMB2 –managementOS
  1. Enable the SMB Multipath vNICs for RDMA
Enable-NetAdapterRDMA -Name “vEthernet (SMB1)”,”vEthernet (SMB2)”
Note: SMB1 and SMB2 should not be attached directly to Virtual Machines. When creating a storage device inside a VM, the access to Storage Spaces Direct volumes will use these RDMA devices. This is managed automatically by Hyper-V.

Next Steps

  • The hyper-converged environment is complete but Virtual Machine workloads cannot yet be deployed.
  • Use the Cluster Manager GUI or PowerShell to deploy and manage virtual machines.

Was this article helpful?

Submit a Comment

Your email address will not be published. Required fields are marked *