Download articleContents
Introduction
This article is the third of a four part series.
- Core Cluster
- Troubleshooting Storage Clusters
- Configuring Storage Network Infrastructures [this article]
- Managing Storage Clusters
In Part 1 Core Cluster the base functions of Windows Storage Spaces Direct was described. A storage infrastructure is complex with multiple hardware and software systems. Once the base functionality has been tested completely, the steps here in Part 3 Network Infrastructure can be completed. Layering the configurations in this way will allow the administrator to better understand and troubleshoot the storage environment before adding additional technologies.
- Configure QoS networking
- Configure the Hyper-V virtual adapters
- Create the Switch Enabled Teaming virtual switch
- Create the SMB Multipath infrastructure
Hyper-v RDMA and SET switch network architecture
The diagram shows the Hyper-V network components and the storage paths for VMs accessing storage.
- Switch Embedded Teaming (SET) if an advanced virtual switch which supports teamed physical network ports as well as SMB Direct, SMB Multi-Channel and RDMA technologies.
- pNIC is RDMA enabled physical ports, for example hardware from (Mellanox, Chelsio, HP, Dell and Intel).
- Storage network traffic uses the SMB protocol. Hyper-V and Storage Spaces work under-the-hood automatically routing RDMA storage access over internal Hyper-V RDNA enabled virtual adaptors. VMs access storage using the local path [C:\StorageCluster\<folder name>.
- VMs access storage on non-local storage nodes across the pNIC physical network LAN.
Configure Network for RDMA
Perform the following procedure separately on each node in your cluster.
Install Feature Prerequisites
Perform this step on each node.
1 | Install-WindowsFeature -Name Data-Center-Bridging -ComputerName HC-Node1 |
Configure QoS policy for SMB Direct
Perform all of the following steps 1-11 in this section on each node.
- Set up a policy to tag SMB-Direct packets with a priority tag.
In this example we use priority tag “3”. Keep in mind that DCB’s QoS policies apply globally, so SMB packets sent using RDMA (“NetDirect” is synonymous with RDMA) will always get tagged with the value “3” no matter what interface they are sent on.
1 | New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3 |
The response should look like:
1 2 3 4 5 6 7 | Name : SMB Owner : Group Policy (Machine) NetworkProfile : All Precedence : 127 JobObject : NetDirectPort : 445 PriorityValue : 3 |
- Enable Priority-based Flow Control (PFC) for the SMB-Direct traffic
1 | Enable-NetQosFlowControl -priority 3 |
If the PFC enablement succeeds there is no response.
- Reserve some bandwidth for the SMB-Direct storage traffic
This example uses 50%, but you may want to reserve less or more depending on what you expect the ratio of storage verses other types of traffic.
1 | New-NetQosTrafficClass "SMB" -priority 3 -bandwidthpercentage 50 -algorithm ETS |
The response to the traffic class creation should look like:
1 2 3 | Name Algorithm Bandwidth(%) Priority PolicySet IfIndex IfAlias ---- --------- ------------ -------- --------- ------- ------- SMB ETS 50 3 Global |
- Set the policies on each physical interface
1 | Enable-NetAdapterQos -InterfaceAlias Provider1,Provider2 |
- Disable Data Center Bridging Exchange (DCBX) traffic
Refer to the Microsoft documentation Managing the Local DCBX Willing State.
1 | Enable- Set-NetQosDcbxSestting Provider1,Provider2 -Willing $False |
- Disable all non-SMB/RDMA traffic goes without a priority tag
1 | New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 0 |
- Disable PFC traffic on the non-SMB traffic
1 | Disable-NetQosFlowControl -priority 0,1,2,4,5,6,7 |
- Test NetQosFlowControl configuration
1 | Get-NetQosFlowControl |
Which should return “True” for the priority tags for which you have turned on PFC and “False” for the rest, e.g.,
1 2 3 4 5 6 7 8 9 10 | Priority Enabled PolicySet IfIndex IfAlias -------- ------- --------- ------- ------- 0 False Global 1 False Global 2 False Global 3 True Global 4 False Global 5 False Global 6 False Global 7 False Global |
- Test the NetAdapterQos configuration
1 | Get-NetAdapterQos -Name Provider1,Provider2 |
which returns:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | Name : Provider1 Enabled : True Capabilities : Hardware Current -------- ------- MacSecBypass : NotSupported NotSupported DcbxSupport : None None NumTCs(Max/ETS/PFC) : 8/8/8 8/8/8 OperationalTrafficClasses : TC TSA Bandwidth Priorities -- --- --------- ---------- 0 ETS 50% 0-2,4-7 1 ETS 50% 3 OperationalFlowControl : Priority 3 Enabled OperationalClassifications : Protocol Port/Type Priority -------- --------- -------- Default 0 NetDirect 445 3 Name : Provider2 Enabled : True Capabilities : Hardware Current -------- ------- MacSecBypass : NotSupported NotSupported DcbxSupport : None None NumTCs(Max/ETS/PFC) : 8/8/8 8/8/8 OperationalTrafficClasses : TC TSA Bandwidth Priorities -- --- --------- ---------- 0 ETS 50% 0-2,4-7 1 ETS 50% 3 OperationalFlowControl : Priority 3 Enabled OperationalClassifications : Protocol Port/Type Priority -------- --------- -------- Default 0 NetDirect 445 3 |
- Enable RDMA on the physical adapters
1 | Enable-NetAdapterRdma Provider1,Provider2 |
The return should look like
1 2 3 4 | Name InterfaceDescription Enabled ---- -------------------- ------- NIC1 Chelsio Network Adapter True NIC2 Chelsio Network Adapter #2 True |
- Confirm that SMB also sees these interfaces as RDMA-capable
1 | Get-SmbClientNetworkInterface |
The result should look like:
1 2 3 4 | Interface Index RSS Capable RDMA Capable Speed IpAddresses --------------- ----------- ------------ ----- ----------- 3 True True 40 Gbps {fe80::e14f:b55:b3dc:b03c, 192.168.1.3} 7 True True 40 Gbps {fe80::9ce6:c07:9aab:d0f4, 192.168.2.3} |
If for some reason the RDMA Capable column in the Get-SmbClientNetworkInterface output shows False, it may require a reboot of the host to get SMB to update the value.
Create the SET vSwitch
- Create the SET switch on each node
1 | New-VMSwitch -Name Provider -NetAdapterName Provider1,Provider2 -AllowManagementOS $true -EnableEmbeddedTeaming $true |
The response should be something like:
1 2 3 | Name SwitchType NetAdapterInterfaceDescription ---- ---------- ------------------------------ Provider External Teamed-Interface |
- Add Hyper-V vNICs to the SET virtual switch
This configures a virtual NIC (vNIC) attached to the virtual switch that you just configured for the management OS to use.
1 | Add-VMNetworkAdapter –SwitchName Provider –Name hvNIC –managementOS |
The hvNIC will be the virtual NIC available to Virtual Machines for client traffic.
Create the SMB-Multipath vNICs
Storage access for Hyper-V VMs will use RDMA using the following vNIC configurations.
- Create the Hyper-V vNICs. Two NICs are created to supply SMB Multipath two paths
1 2 | Add-VMNetworkAdapter –SwitchName Provider –Name SMB1 –managementOS Add-VMNetworkAdapter –SwitchName Provider –Name SMB2 –managementOS |
1 2 | Get-VMNetworkAdapter -ManagementOS SMB1 | Set-VMNetworkAdapterVLan -Access -VlanID 20 Get-VMNetworkAdapter -ManagementOS SMB2 | Set-VMNetworkAdapterVLan -Access -VlanID 20 |
- Enable the SMB Multipath vNICs for RDMA
1 | Enable-NetAdapterRDMA -Name “vEthernet (SMB1)”,”vEthernet (SMB2)” |
Next Steps
- The hyper-converged environment is complete and Virtual Machine workloads can be deployed.
- Use the Cluster Manager GUI or PowerShell to deploy and manage virtual machines.
- Create storage volumes, rename the new folder, then use the direct volume folder path:
C:\ClusterStorage\<volume name>
Hello
I tested this artical with Mellanox ConnectX-3 IPoIB Adapter.
“Step 4 Set the policies on each physical interface”
PS: Enable-NetAdapterQos -InterfaceAlias Provider1
response is
Enable-NetAdapterQos : ‘Provider1 ‘과(와) 같은 ‘Name’ 속성을 가진 MSFT_NetAdapterQosSettingData 개체가 없습니다. 속성 값을 검증하고 다시 시도하십시오.
위치 줄:1 문자:1
+ Enable-NetAdapterQos -InterfaceAlias “Provider1 ”
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (Provider1 :String) [Enable-NetAdapterQos], CimJobException
+ FullyQualifiedErrorId : CmdletizationQuery_NotFound_Name,Enable-NetAdapterQos
I can’t find any solution for this error message.
Help me PLZ
Thank you very much for the information you shared, it’s all I’ve been looking for
I think this is an informative post and it is very useful and knowledgeable. therefore, I would like to thank you for the efforts you have made in writing this article.