Argon Systems has developed the ultimate high-performance storage solution using Storage Spaces Direct on Windows Server 2016. It is an all flash array configured with NVDIMM and NVMe hardware and high speed networking components.
Hyper-converged storage technology has been focused on a converged compute and storage solution in a single server chassis. The convergence of compute and storage, or hypervisor and storage converged solutions, has benefits where Virtual Machines and storage is combined on into a single server. This provides price and management benefits, especially in the small to mid-sized business market. More often we at Argon are deploying Windows 2016 Storage Spaces Direct as a storage only solution. The additional storage features in Windows 2016 Storage Spaces Direct make an amazingly feature rich and high-performance storage infrastructure solution.
Storage solutions for this discussion can be segmented into three categories:
- Archival Storage – This is a large or massive storage infrastructure which is typically focused on massive amounts of mostly static storage. This might include backup data sets, large media assets or other similar types of data
- Application Storage – This is a combination of high performance and large capacity storage used for hosting VMs and typical applications such as ERP systems and SQL databases
- High-Performance Storage – Typically all Flash Storage systems for high speed low latency storage requirements such as high volume transactions such as SQL Server infrastructures
We will describe how Windows 2016 Storage Spaces Direct creates a solution for High-Performance Flash Only Storage.
The main technologies used in this storage solution include:
- High-Speed Low-Latency Ethernet network spine
- Servers with high-end motherboards with multiple PCIe V3 (or V4 when available) bus networks
- Servers which support multiple U.2 NVMe SSD form factor drives
- Servers with a motherboard which supports multiple NVDIMM-N memory (SCM – Storage Class Memory) drives
In this example, we are building the system using Microsoft Windows Server 2016, Datacenter version, and the following components.
|Ultimate DB Cloud Server|
|24 Intel® P3700 NVMe U.2 drives|
| NVDIMM-N Micron Storage Class Memory Drives
Hyper-converged Storage Advantages
Converged and previous hyper-converged storage technologies required shared storage. This shared storage systems often were composed JBOD enclosures with SAS networks or other technologies such as Fiber Channel. These technologies multiplex multiple drives over a single network channel and are not appropriate for an all-flash storage array architecture.
Hyper-converged storage attach drives directly to the server bypassing the shared storage infrastructure. Hyper-converged storage is often called “shared nothing” infrastructures since each server only uses local storage without JBODs, SAS, FiberChannel or any of the older storage technologies.
Hyper-converged servers are stacked into a cluster of servers. These servers together provide a highly available scale-out file server. Data is replicated among servers and access to storage across servers is through the Ethernet backbone or “spine”. This storage network must have the best performance as possible so as not to be a bottleneck. For this exercise, we are using high-speed networking technology which will allow the storage system to scale simply be adding additional servers.
With disk systems using NVDIMM and NVMe which perform at the speeds of memory, drives must be attached directly to the PCIe bus or directly to the memory bus. With hyper-converged technologies from Microsoft bundled with high performance networks, we can construct high performance storage systems from commodity components which until very recently were only possible with high-end storage systems using exotic purpose built components.
Storage Logical Architecture
In the diagram below, we are using four servers, each configured with the same number of NVDIMM-N drives and NVMe drives.
Each server has four 8GB NVDIMM-N drives and 24 2TB NVMe.
In this simple example, we will have two pools, a Very Fast Pool with only NVDIMM drives and one pool with only NVMe drives. When we display a disk list, we will see 16 NVDIMM drives and 96 NVMe drives.
- The raw storage for NVDIMM is 128GB
- The raw storage for NVMe drives is 192TB
This storage could be increased by adding additional servers to the cluster. Storage performance increases linearly with the addition of storage servers. Shared storage solutions lose do not scale in a linear fashion due the shared storage creates a performance bottleneck.
Storage Network Technology
Each server is attached via the Mellanox Network Interface Cards (NIC). Microsoft has supported RDMA for over ten years and is years ahead of other technology vendors in RDMA. Mellanox network cards support RDMA and transports RDMA data over the ROCE transport protocol.
Microsoft uses the SMBv3 protocol as the SAN storage protocol, which uses RDMA which allows Hyper-V Virtual Machines to communicate directly to the storage infrastructure. Virtual Machines then have a high speed low latency storage infrastructure to communicate.
Storage Servers also use the storage network to communicate among themselves. This allows data to be shared and replicated at high speeds among the storage server disk systems. This is especially important with high speed drives such as NVDIMM and NVMe.
In this example, we are using Mellanox SNxxxx model switches. These switches have evolved from InfiniBand and use InfiniBand ASIC silicon technologies. This allows Ethernet to work at wire speed, or processed in the chips without having to be processed in software. The network latency in these networks are critical and must be as low as possible.
SQL Server Layout
There are many uses for high speed storage such as analytics and other applications which rely on storage for performance. High speed storage accelerates applications. SQL Server and other databases are a common application which benefits from high performance storage. Storage is often the ultimate bottleneck of databases and databases are the ultimate bottleneck for application performance, so high performance storage can greatly benefit critical business applications.
We will discuss briefly how SQL can benefit from this Hyper-converged all-flash storage solution and how the table files are placed on storage.
We created two storage pools, one for NVDIMM and one for NVMe. The NVDIMM are the fastest storage available today and NVMe are used for fast but larger capacity storage. SQL Databases process transactions and guarantee data integrity by writing a transaction log record before each group of data records are written and then a transaction record after the data records. This process guarantees data integrity. If a database is interrupted while data is being written, when the database is restarted, the transactions which were incomplete as per the transaction log are reversed out. The database is then in a consistent state and application data integrity is guaranteed.
The records written to the transaction log are short records, but the number of records written are high. SQL Performance is greatly accelerated if the SQL Transaction Log is placed on the highest performing storage available.
The storage layout for SQL Server tables is shown below.
The SQL Transaction Log is placed on the volume created on the NVDIMM pool of disks.
The other database files including the System Tables, Data Tables and tempDB are placed on the main capacity storage NVMe drives.
For the example shown, the storage for NVDIMM is shared on the four servers and mirrored 2 or 3 times for high availability. The same for the database files on the NVMe drives.
The Storage Spaces Direct handles the resiliency and has high-end storage features recently only available on large expensive SAN technologies.
Storage Spaces Direct combined with the high speed hardware components described in this article provides storage features and performance which can surpass all but the most exotic storage technologies.