Configuring Storage Spaces Direct Step by Step: Part 3 Network Infrastructure

Contents

1 Introduction
2 Configure Network for RDMA
3 Install Feature Prerequisites
4 Configure QoS policy for SMB Direct
5 Create the SET vSwitch
6 Create the SMB-Multipath vNICs
7 Next Steps

Introduction

This article is the third of a four part series.

Core Cluster
Troubleshooting Storage Clusters
Configuring Storage Network Infrastructures [this article]
Managing Storage Clusters

In Part 1 Core Cluster the base functions of Windows Storage Spaces Direct was described. A storage infrastructure is complex with multiple hardware and software systems. Once the base functionality has been tested completely, the steps here in Part 3 Network Infrastructure can be completed. Layering the configurations in this way will allow the administrator to better understand and troubleshoot the storage environment before adding additional technologies.

Configure QoS networking
Configure the Hyper-V virtual adapters
Create the Switch Enabled Teaming virtual switch
Create the SMB Multipath infrastructure

Hyper-v RDMA and SET switch network architecture

The diagram shows the Hyper-V network components and the storage paths for VMs accessing storage.

Switch Embedded Teaming (SET) if an advanced virtual switch which supports teamed physical network ports as well as SMB Direct, SMB Multi-Channel and RDMA technologies.
pNIC is RDMA enabled physical ports, for example hardware from (Mellanox, Chelsio, HP, Dell and Intel).
Storage network traffic uses the SMB protocol. Hyper-V and Storage Spaces work under-the-hood automatically routing RDMA storage access over internal Hyper-V RDNA enabled virtual adaptors. VMs access storage using the local path [C:\StorageCluster\<folder name>.
VMs access storage on non-local storage nodes across the pNIC physical network LAN.

Configure Network for RDMA

Perform the following procedure separately on each node in your cluster.

Install Feature Prerequisites

Perform this step on each node.

Note: Not required for servers with iWarp network adapters.

Install-WindowsFeature -Name Data-Center-Bridging -ComputerName HC-Node1

1	Install-WindowsFeature -Name Data-Center-Bridging -ComputerName HC-Node1

Configure QoS policy for SMB Direct

Perform all of the following steps 1-11 in this section on each node.

Set up a policy to tag SMB-Direct packets with a priority tag.

In this example we use priority tag “3”. Keep in mind that DCB’s QoS policies apply globally, so SMB packets sent using RDMA (“NetDirect” is synonymous with RDMA) will always get tagged with the value “3” no matter what interface they are sent on.

Note: While this guide uses the tag value 3, any tag value between 1 and 7 inclusive can be used as long as it is used everywhere through the network in both the hosts and the switches/routers.

New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3

1	New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3

The response should look like:

Name : SMB
Owner : Group Policy (Machine)
NetworkProfile : All
Precedence : 127
JobObject :
NetDirectPort : 445
PriorityValue : 3

Name : SMB

Owner : Group Policy (Machine)

NetworkProfile : All

Precedence : 127

JobObject :

NetDirectPort : 445

PriorityValue : 3

Enable Priority-based Flow Control (PFC) for the SMB-Direct traffic

Enable-NetQosFlowControl -priority 3

1	Enable-NetQosFlowControl -priority 3

If the PFC enablement succeeds there is no response.

Reserve some bandwidth for the SMB-Direct storage traffic

This example uses 50%, but you may want to reserve less or more depending on what you expect the ratio of storage verses other types of traffic.

New-NetQosTrafficClass "SMB" -priority 3 -bandwidthpercentage 50 -algorithm ETS

1	New-NetQosTrafficClass "SMB" -priority 3 -bandwidthpercentage 50 -algorithm ETS

The response to the traffic class creation should look like:

Name Algorithm Bandwidth(%) Priority PolicySet IfIndex IfAlias
---- --------- ------------ -------- --------- ------- -------
SMB  ETS       50           3        Global

Name Algorithm Bandwidth(%) Priority PolicySet IfIndex IfAlias

---- --------- ------------ -------- --------- ------- -------

SMB ETS 50 3 Global

Set the policies on each physical interface

Enable-NetAdapterQos -InterfaceAlias Provider1,Provider2

1	Enable-NetAdapterQos -InterfaceAlias Provider1,Provider2

Disable Data Center Bridging Exchange (DCBX) traffic

Refer to the Microsoft documentation Managing the Local DCBX Willing State.

Note: This is not used by iWarp

Enable- Set-NetQosDcbxSestting Provider1,Provider2 -Willing $False

1	Enable- Set-NetQosDcbxSestting Provider1,Provider2 -Willing $False

Disable all non-SMB/RDMA traffic goes without a priority tag

New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 0

1	New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 0

Disable PFC traffic on the non-SMB traffic

Disable-NetQosFlowControl -priority 0,1,2,4,5,6,7

1	Disable-NetQosFlowControl -priority 0,1,2,4,5,6,7

Test NetQosFlowControl configuration

Get-NetQosFlowControl

1	Get-NetQosFlowControl

Which should return “True” for the priority tags for which you have turned on PFC and “False” for the rest, e.g.,

Priority Enabled PolicySet IfIndex IfAlias
-------- ------- --------- ------- -------
0        False   Global
1        False   Global
2        False   Global
3        True    Global
4        False   Global
5        False   Global
6        False   Global
7        False   Global

Priority Enabled PolicySet IfIndex IfAlias

-------- ------- --------- ------- -------

0 False Global

1 False Global

2 False Global

3 True Global

4 False Global

5 False Global

6 False Global

7 False Global

Test the NetAdapterQos configuration

Get-NetAdapterQos -Name Provider1,Provider2

1	Get-NetAdapterQos -Name Provider1,Provider2

which returns:

Name : Provider1
Enabled : True
Capabilities :                                     Hardware Current
                                                   -------- -------
                             MacSecBypass        : NotSupported NotSupported
                             DcbxSupport         : None None
                             NumTCs(Max/ETS/PFC) : 8/8/8 8/8/8
OperationalTrafficClasses  : TC TSA Bandwidth Priorities
                             -- --- --------- ----------
                              0 ETS 50%       0-2,4-7
                              1 ETS 50%       3
OperationalFlowControl     : Priority 3 Enabled
OperationalClassifications : Protocol Port/Type Priority
                             -------- --------- --------
                             Default            0
                             NetDirect 445      3
Name                       : Provider2
Enabled                    : True
Capabilities               :                       Hardware     Current
                                                   --------     -------
                             MacSecBypass        : NotSupported NotSupported
                             DcbxSupport         : None         None
                             NumTCs(Max/ETS/PFC) : 8/8/8        8/8/8
OperationalTrafficClasses  : TC TSA Bandwidth Priorities
                             -- --- --------- ----------
                              0 ETS 50%       0-2,4-7
                              1 ETS 50%       3
OperationalFlowControl     : Priority 3 Enabled
OperationalClassifications : Protocol  Port/Type Priority
                             --------  --------- --------
                             Default   0
                             NetDirect 445 3

Name : Provider1

Enabled : True

Capabilities : Hardware Current

-------- -------

MacSecBypass : NotSupported NotSupported

DcbxSupport : None None

NumTCs(Max/ETS/PFC) : 8/8/8 8/8/8

OperationalTrafficClasses : TC TSA Bandwidth Priorities

-- --- --------- ----------

0 ETS 50% 0-2,4-7

1 ETS 50% 3

OperationalFlowControl : Priority 3 Enabled

OperationalClassifications : Protocol Port/Type Priority

-------- --------- --------

Default 0

NetDirect 445 3

Name : Provider2

Enabled : True

Capabilities : Hardware Current

-------- -------

MacSecBypass : NotSupported NotSupported

DcbxSupport : None None

NumTCs(Max/ETS/PFC) : 8/8/8 8/8/8

OperationalTrafficClasses : TC TSA Bandwidth Priorities

-- --- --------- ----------

0 ETS 50% 0-2,4-7

1 ETS 50% 3

OperationalFlowControl : Priority 3 Enabled

OperationalClassifications : Protocol Port/Type Priority

-------- --------- --------

Default 0

NetDirect 445 3

Enable RDMA on the physical adapters

Enable-NetAdapterRdma Provider1,Provider2

1	Enable-NetAdapterRdma Provider1,Provider2

The return should look like

Name InterfaceDescription        Enabled
---- --------------------        -------
NIC1 Chelsio Network Adapter     True
NIC2 Chelsio Network Adapter #2  True

Name InterfaceDescription Enabled

---- -------------------- -------

NIC1 Chelsio Network Adapter True

NIC2 Chelsio Network Adapter #2 True

Confirm that SMB also sees these interfaces as RDMA-capable

Get-SmbClientNetworkInterface

1	Get-SmbClientNetworkInterface

The result should look like:

Interface Index RSS Capable RDMA Capable Speed   IpAddresses
--------------- ----------- ------------ -----   -----------
3               True        True         40 Gbps {fe80::e14f:b55:b3dc:b03c, 192.168.1.3}
7               True        True         40 Gbps {fe80::9ce6:c07:9aab:d0f4, 192.168.2.3}

Interface Index RSS Capable RDMA Capable Speed IpAddresses

--------------- ----------- ------------ ----- -----------

3 True True 40 Gbps {fe80::e14f:b55:b3dc:b03c, 192.168.1.3}

7 True True 40 Gbps {fe80::9ce6:c07:9aab:d0f4, 192.168.2.3}

If for some reason the RDMA Capable column in the Get-SmbClientNetworkInterface output shows False, it may require a reboot of the host to get SMB to update the value.

Create the SET vSwitch

Create the SET switch on each node

New-VMSwitch -Name Provider -NetAdapterName Provider1,Provider2 -AllowManagementOS $true -EnableEmbeddedTeaming $true

1	New-VMSwitch -Name Provider -NetAdapterName Provider1,Provider2 -AllowManagementOS $true -EnableEmbeddedTeaming $true

The response should be something like:

Name     SwitchType NetAdapterInterfaceDescription
----     ---------- ------------------------------
Provider External   Teamed-Interface

Name SwitchType NetAdapterInterfaceDescription

---- ---------- ------------------------------

Provider External Teamed-Interface

Add Hyper-V vNICs to the SET virtual switch

This configures a virtual NIC (vNIC) attached to the virtual switch that you just configured for the management OS to use.

Note: Because SET is integrated into the Hyper-V Virtual Switch, you cannot use SET Switch inside of a virtual machine.

Add-VMNetworkAdapter –SwitchName Provider –Name hvNIC –managementOS

1	Add-VMNetworkAdapter –SwitchName Provider –Name hvNIC –managementOS

The hvNIC will be the virtual NIC available to Virtual Machines for client traffic.

Create the SMB-Multipath vNICs

Storage access for Hyper-V VMs will use RDMA using the following vNIC configurations.

Create the Hyper-V vNICs. Two NICs are created to supply SMB Multipath two paths

Add-VMNetworkAdapter –SwitchName Provider –Name SMB1 –managementOS
Add-VMNetworkAdapter –SwitchName Provider –Name SMB2 –managementOS

1 2	Add-VMNetworkAdapter –SwitchName Provider –Name SMB1 –managementOS Add-VMNetworkAdapter –SwitchName Provider –Name SMB2 –managementOS

Note: Some network switches and iWarp may require VLANs. The following sets a VLAN.

Get-VMNetworkAdapter -ManagementOS SMB1 | Set-VMNetworkAdapterVLan -Access -VlanID 20
Get-VMNetworkAdapter -ManagementOS SMB2 | Set-VMNetworkAdapterVLan -Access -VlanID 20

1 2	Get-VMNetworkAdapter -ManagementOS SMB1 \| Set-VMNetworkAdapterVLan -Access -VlanID 20 Get-VMNetworkAdapter -ManagementOS SMB2 \| Set-VMNetworkAdapterVLan -Access -VlanID 20

Enable the SMB Multipath vNICs for RDMA

Enable-NetAdapterRDMA -Name “vEthernet (SMB1)”,”vEthernet (SMB2)”

1	Enable-NetAdapterRDMA -Name “vEthernet (SMB1)”,”vEthernet (SMB2)”

Note: SMB1 and SMB2 should not be attached directly to Virtual Machines. When creating a storage device inside a VM, the access to Storage Spaces Direct volumes will use these RDMA devices. This is managed automatically by Hyper-V.

Next Steps

The hyper-converged environment is complete and Virtual Machine workloads can be deployed.
Use the Cluster Manager GUI or PowerShell to deploy and manage virtual machines.
Create storage volumes, rename the new folder, then use the direct volume folder path:
C:\ClusterStorage\<volume name>

1 Comment

Jinwoo Kim on Mar 20, 2019 at 1:29 AM

Hello
I tested this artical with Mellanox ConnectX-3 IPoIB Adapter.

“Step 4 Set the policies on each physical interface”

PS: Enable-NetAdapterQos -InterfaceAlias Provider1

response is

Enable-NetAdapterQos : ‘Provider1 ‘과(와) 같은 ‘Name’ 속성을 가진 MSFT_NetAdapterQosSettingData 개체가 없습니다. 속성 값을 검증하고 다시 시도하십시오.
위치 줄:1 문자:1
+ Enable-NetAdapterQos -InterfaceAlias “Provider1 ”
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (Provider1 :String) [Enable-NetAdapterQos], CimJobException
+ FullyQualifiedErrorId : CmdletizationQuery_NotFound_Name,Enable-NetAdapterQos

I can’t find any solution for this error message.

Help me PLZ

Configuring Storage Spaces Direct Step by Step: Part 3 Network Infrastructure

Introduction

Configure Network for RDMA

Install Feature Prerequisites

Configure QoS policy for SMB Direct

Create the SET vSwitch

Create the SMB-Multipath vNICs

Next Steps

1 Comment

Submit a Comment Cancel reply

FREE Consultation

Hybrid Cloud Building Blocks

Technical Articles Topics

Latest Articles from Argon

Popular Articles from Argon

Configuring Storage Spaces Direct Step by Step: Part 3 Network Infrastructure

Introduction

Configure Network for RDMA

Install Feature Prerequisites

Configure QoS policy for SMB Direct

Create the SET vSwitch

Create the SMB-Multipath vNICs

Next Steps

Related Articles

1 Comment

Submit a Comment Cancel reply

FREE Consultation

Hybrid Cloud Building Blocks

Technical Articles Topics

Latest Articles from Argon

Popular Articles from Argon