In large scale AI clusters storage is responsible for delivering training data fast enough to keep accelerators busy and completing checkpoint saves and recoveries quickly enough to avoid costly stalls.

MLPerf Storage is an industry-standard benchmark developed by MLCommons that reproduces the I/O patterns of different stages of AI pipelines covering training, checkpointing, context cache and vector database scenarios, reporting how efficiently a storage system can sustain the I/O rates required by the workload maximizing accelerators throughput without introducing bottlenecks.

Azure submitted results in the MLPerf Storage v3.0 round, and its training and checkpointing results rank as the top submissions among major hyperscalers, in both demonstrated scale and sustained performance.

In checkpointing, Azure Managed Lustre reached 642.2 GiB/s of write throughput and 489.6 GiB/s during recovery, showing the highest peak write performance for both the 405B and 1T model configurations among hyperscalers and public cloud submissions. In training, Azure Managed Lustre demonstrated the ability to sustain the required accelerator utilization with hundreds of GiB/s across five different tiers and file systems ranging from 40 TiB to the largest system presented in the submission, a 25 PiB file system in a single POSIX namespace.

These peer-reviewed results demonstrate that Azure Managed Lustre is a service built for the scale AI now demands and that can sustain the storage performance required to keep AI accelerators highly utilized across different stages of AI workloads.

What the Benchmark Measures

MLPerf Storage answers one question: how effectively can a storage system keep AI accelerators highly utilized and operating efficiently?

In a modern AI cluster, accelerators are among the most constrained and expensive resources. Every second they wait for training samples or checkpoint I/O reduces the productive value of provisioned compute. MLPerf Storage isolates the storage path and measures whether it can sustain the I/O rate required by a representative AI workload.

The benchmark has an OPEN and a CLOSED division. In the CLOSED division, workload code and benchmark parameters are standardized, while submitters may tune the storage system within the benchmark rules. This constraint reduces variation outside the storage path and makes results more directly comparable across submitters. Azure submitted every reported result in CLOSED division.

Training category

The training category evaluates whether a storage system can sustain the data-delivery rate required to keep simulated accelerators utilized throughout an extended time window, rather than only demonstrating a short burst of peak throughput.

Why it matters: accelerators represent the widest portion of an AI cluster’s cost, and storage stalls reduce the productive value of provisioned compute. Because the benchmark evaluates simulated accelerator utilization rather than raw bandwidth alone, a passing score indicates that the storage path sustained the required workload’s data-delivery requirements without becoming a bottleneck.

Checkpointing category

The checkpointing category measures how quickly a storage system can write model state at a synchronization barrier and read that state during recovery, including recovery on different client hosts. These operations model the storage demands involved when a distributed training job resumes after a failure.

Why it matters: checkpoint writes pause progress across a distributed training job, making checkpoint duration a shared overhead for the cluster with accelerator stalling. Faster writes let teams checkpoint more often and reduce the work at risk without significantly affecting accelerator utilization, while faster recovery reads shorten the time required to resume a disrupted job.

What Was Tested

Every result used Azure Managed Lustre, a managed parallel file system that presents clients with a single POSIX namespace. Testing covered five service tiers and file-system capacities from 40 TiB to approximately 25 PiB. MLPerf Storage generates workload I/O from standard client nodes rather than GPU nodes, and the submission used publicly available Azure virtual machine SKUs, network configurations, deployment patterns, and service interfaces.

Results were produced across multiple Azure regions using configurations available to customers today. Together with the published system descriptions and configuration details, these results provide a practical reference for evaluating Azure Managed Lustre performance.

In the training category, Azure Managed Lustre sustained throughput and maintained simulated accelerator utilization above 90% for five consecutive hours on the UNET3D workload with 13 distinct system configurations. The submitted configurations ranged from tens of TiB to tens of PiB and delivered sustained throughput in the hundreds of GiB/s. The five-hour result demonstrates predictable storage performance over an extended benchmark run rather than a short-lived burst. AMLFS delivers target performance across all service tiers, enabling customers to optimize capacity and performance for each workload without compromising accelerator utilization, as shown in Figure 1.

Figure 1, AMLFS UNET3D Training performance (B200) across different bandwidth density configurations

Most relevant to highlight is how Azure Managed Lustre delivers at more than 96% the nominal bandwidth density across all tiers tested in training category, and how the results, taking AMLFS 125 as a tier, scale almost linearly increasing the number of accelerators (Figure 2).

Figure 2, AMLFS UNET3D Training performance (B200) delivered bandwidth density and scalability.

In checkpointing category, the benchmark covered four Llama3 scales, from an 8B model with a 105 GB checkpoint to a 1T-class model with an 18 TB checkpoint, and Azure submitted results for all four. Within those benchmark rules, Azure Managed Lustre reached up to 642 GiB/s of checkpoint write throughput and up to 490 GiB/s during recovery, enabling the tested checkpoints to be written and restored in tens of seconds or less. For checkpointing, AMLFS 125 has been picked as example tier, but similar results are expected, as demonstrated in the training scenario, with different AMLFS tiers.

Figure 3, AMLFS checkpointing performance across different model size.

The published results give the AI community a peer-reviewed view of Azure Managed Lustre under modern AI workloads, including sustained throughput, predictable performance, and deployment choices across service tiers. Full system descriptions and performance numbers are available in the MLCommons MLPerf Storage v3.0 results tables.

Five Tiers, 20 to 500 MB/s per TiB, One Right Size for Every AI Workload

Results span five Azure Managed Lustre service tiers, from 20 MB/s per TiB to 500 MB/s per TiB, showing how the tested performance extends across the range of configurations available for demanding AI workloads.

That breadth is deliberate because AI infrastructure requirements vary by model size, accelerator count, data volume, and I/O pattern; there is no single correct storage configuration. Results across five tiers show how customers can balance throughput, capacity, and cost as models, datasets, and clusters grow. Correct sizing can reduce storage bottlenecks and shorten time to results while retaining the durability and resiliency of a managed cloud file system.

Getting Started with Azure Managed Lustre

To evaluate Azure Managed Lustre for an AI or HPC workload, review the Azure Managed Lustre documentation for service capabilities, planning guidance, and deployment options. Then use the Azure portal deployment guide to configure and create a file system.

For deployment examples, explore the AI Infrastructure on Azure GitHub repository. It provides reference architectures, best practices, and templates for AI and HPC environments, including examples built with Azure CycleCloud and Azure Kubernetes Service. Together with the resources below, these examples show how Azure Managed Lustre can serve as a high-performance shared storage layer for AI workloads.

Learn More

Azure Resources

MLPerf Resources

This article was originally published by Microsoft's Azure High Performance Computing Blog. You can find the original article here.

Tags: Artificial Intelligence, Backup/Recovery, Best Practice, Cluster, Storage, Virtual Machine [VM]

Scaling AI Storage: Azure Managed Lustre excels in MLPerf Storage v3.0