Select Page

Ultimate Performance for Database Storage with Windows Server 2016 and Hyper-converged Infrastructure

Argon Systems has developed the ultimate high-performance storage solution using Storage Spaces Direct on Windows Server 2016. It is an all flash array configured with NVDIMM and NVMe hardware and high speed networking components.

Hyper-converged storage technology has been focused on a converged compute and storage solution in a single server chassis. The convergence of compute and storage, or hypervisor and storage converged solutions, has benefits where Virtual Machines and storage is combined on into a single server. This provides price and management benefits, especially in the small to mid-sized business market. More often we at Argon are deploying Windows 2016 Storage Spaces Direct as a storage only solution. The additional storage features in Windows 2016 Storage Spaces Direct make an amazingly feature rich and high-performance storage infrastructure solution.

Storage solutions for this discussion can be segmented into three categories:

  • Archival Storage – This is a large or massive storage infrastructure which is typically focused on massive amounts of mostly static storage. This might include backup data sets, large media assets or other similar types of data
  • Application Storage – This is a combination of high performance and large capacity storage used for hosting VMs and typical applications such as ERP systems and SQL databases
  • High-Performance Storage – Typically all Flash Storage systems for high speed low latency storage requirements such as high volume transactions such as SQL Server infrastructures

We will describe how Windows 2016 Storage Spaces Direct creates a solution for High-Performance Flash Only Storage.

Technology Components

The main technologies used in this storage solution include:

  • High-Speed Low-Latency Ethernet network spine
  • Servers with high-end motherboards with multiple PCIe V3 (or V4 when available) bus networks
  • Servers which support multiple U.2 NVMe SSD form factor drives
  • Servers with a motherboard which supports multiple NVDIMM-N memory (SCM – Storage Class Memory) drives

In this example, we are building the system using Microsoft Windows Server 2016, Datacenter version, and the following components.

SuperServer 2028U-TRT+ Ultimate DB Cloud Server
  • HPE and Dell have similar systems but at the time of this writing have limited support for NVMe, though these servers can support 6 NVMe drives and 16 SATA SSD drives
ConnectX-4 Single Dual-Port Adapter Mellanox ConnectX®-4
  • Dual 50GbE or 100GbE ports
  • Supports RDMA using the ROCE low latency Ethernet protocol
  • ROCE protocol allows VMs to communicate directly with Storage bypassing the overhead of IP
Intel P3700 NVMe U.2 drive 24 Intel® P3700 NVMe U.2 drives
  • These drives are available in sizes from 200GB to 4TB
  • NVMe drives connect to the PCIe bus, bypassing the SATA infrastructure greatly increasing IO performance and reducing latency
  • Hot swappable drives in an SSD 2.5” form factor
 NVDIMM-N Micron Storage Class Memory Drives
  • NVDIMM-N drives connect directly to the memory bus, further bypassing the PCIe bus and processing IO operations at the speed of memory
  • 4x 8GB NVDIMM drives per server (32GB NVDIMM modules have been announced)

Hyper-converged Storage Advantages

Converged and previous hyper-converged storage technologies required shared storage. This shared storage systems often were composed JBOD enclosures with SAS networks or other technologies such as Fiber Channel. These technologies multiplex multiple drives over a single network channel and are not appropriate for an all-flash storage array architecture.

Hyper-converged storage attach drives directly to the server bypassing the shared storage infrastructure. Hyper-converged storage is often called “shared nothing” infrastructures since each server only uses local storage without JBODs, SAS, FiberChannel or any of the older storage technologies.

Hyper-converged servers are stacked into a cluster of servers. These servers together provide a highly available scale-out file server. Data is replicated among servers and access to storage across servers is through the Ethernet backbone or “spine”. This storage network must have the best performance as possible so as not to be a bottleneck. For this exercise, we are using high-speed networking technology which will allow the storage system to scale simply be adding additional servers.

With disk systems using NVDIMM and NVMe which perform at the speeds of memory, drives must be attached directly to the PCIe bus or directly to the memory bus. With hyper-converged technologies from Microsoft bundled with high performance networks, we can construct high performance storage systems from commodity components which until very recently were only possible with high-end storage systems using exotic purpose built components.

Storage Logical Architecture

Windows Storage Spaces is configured by creating disk pools. In this example we will create two pools, one pool for the NVDIMM drives and one pool for the NVMe drives.

Note: Windows Storage Spaces allow for other disk architectures such as SATA SSD drives or SATA HDD drives. This example is simplified to focus on the flash aspect though other configurations are also possible

In the diagram below, we are using four servers, each configured with the same number of NVDIMM-N drives and NVMe drives.

storage spaces logical layout

Each server has four 8GB NVDIMM-N drives and 24 2TB NVMe.

Note: When using Storage Spaces Direct (S2D), each server can see the drives attached to every server as if the drives were local. For example, when showing a list of NVDIMM drives on any server, the list will show 16 NVDIMM drives as if these drives were local to each server.

In this simple example, we will have two pools, a Very Fast Pool with only NVDIMM drives and one pool with only NVMe drives. When we display a disk list, we will see 16 NVDIMM drives and 96 NVMe drives.

  • The raw storage for NVDIMM is 128GB
  • The raw storage for NVMe drives is 192TB

This storage could be increased by adding additional servers to the cluster. Storage performance increases linearly with the addition of storage servers. Shared storage solutions lose do not scale in a linear fashion due the shared storage creates a performance bottleneck.

Note: There are other ways to create a pool layout, where NVDIMM and NVMe drives could be combined into a pool where the NVDIMM could be used for caching. This pool layout was chosen to be used for SQL storage where the NVDIMM is dedicated to the SQL Transaction Log.

Storage Network Technology

Each server is attached via the Mellanox Network Interface Cards (NIC). Microsoft has supported RDMA for over ten years and is years ahead of other technology vendors in RDMA. Mellanox network cards support RDMA and transports RDMA data over the ROCE transport protocol.

Microsoft uses the SMBv3 protocol as the SAN storage protocol, which uses RDMA which allows Hyper-V Virtual Machines to communicate directly to the storage infrastructure. Virtual Machines then have a high speed low latency storage infrastructure to communicate.

Storage Servers also use the storage network to communicate among themselves. This allows data to be shared and replicated at high speeds among the storage server disk systems. This is especially important with high speed drives such as NVDIMM and NVMe.

smb rmda roce 100gb wire speed technology

In this example, we are using Mellanox SNxxxx model switches. These switches have evolved from InfiniBand and use InfiniBand ASIC silicon technologies. This allows Ethernet to work at wire speed, or processed in the chips without having to be processed in software. The network latency in these networks are critical and must be as low as possible.

SQL Server Layout

There are many uses for high speed storage such as analytics and other applications which rely on storage for performance. High speed storage accelerates applications. SQL Server and other databases are a common application which benefits from high performance storage. Storage is often the ultimate bottleneck of databases and databases are the ultimate bottleneck for application performance, so high performance storage can greatly benefit critical business applications.

We will discuss briefly how SQL can benefit from this Hyper-converged all-flash storage solution and how the table files are placed on storage.

We created two storage pools, one for NVDIMM and one for NVMe. The NVDIMM are the fastest storage available today and NVMe are used for fast but larger capacity storage. SQL Databases process transactions and guarantee data integrity by writing a transaction log record before each group of data records are written and then a transaction record after the data records. This process guarantees data integrity. If a database is interrupted while data is being written, when the database is restarted, the transactions which were incomplete as per the transaction log are reversed out. The database is then in a consistent state and application data integrity is guaranteed.

The records written to the transaction log are short records, but the number of records written are high. SQL Performance is greatly accelerated if the SQL Transaction Log is placed on the highest performing storage available.

The storage layout for SQL Server tables is shown below.

sql server storage spaces logical layout

The SQL Transaction Log is placed on the volume created on the NVDIMM pool of disks.

The other database files including the System Tables, Data Tables and tempDB are placed on the main capacity storage NVMe drives.

For the example shown, the storage for NVDIMM is shared on the four servers and mirrored 2 or 3 times for high availability. The same for the database files on the NVMe drives.


The Storage Spaces Direct handles the resiliency and has high-end storage features recently only available on large expensive SAN technologies.

Storage Spaces Direct combined with the high speed hardware components described in this article provides storage features and performance which can surpass all but the most exotic storage technologies.

Was this article helpful?

Submit a Comment

Your email address will not be published. Required fields are marked *