By Mark Gitau and Azin Heidarshenas
In the MLPerf Inference v6.1 round, Azure demonstrated strong Deepseek-R1 performance on GB300 NVL72 and GB200 NVL72 platforms. Azure submitted GB200 results at 18-node (1-rack) and 72-node (4-rack) scale, and GB300 results at 18 nodes (1 rack), marking our first rack-scale and cross-rack MLPerf Inference submissions. According to results published by MLCommons, Azure achieved the highest reported Interactive score on GB300 NVL72 and the highest reported Server and Offline scores on GB200 NVL72 at both 1- and 4-rack scale, demonstrating the ability to sustain high throughput across latency-constrained and throughput-optimized deployment scenarios.
DeepSeek-R1 is a large Mixture-of-Experts (MoE) model, making it an ideal benchmark for modern AI infrastructure. It stresses compute, networking (both NVLink and scale-out network), memory, and serving infrastructure under realistic large-scale deployment conditions. As a result, performance is determined by the effectiveness of the entire system stack rather than GPU capability alone.
MLPerf Inference evaluates system performance across several deployment scenarios. Offline measures maximum throughput for batch-oriented workloads with no latency constraints. Server simulates production online serving, where requests arrive continuously and systems must maximize throughput while meeting latency targets, evaluating how efficiently a system balances responsiveness and utilization. Interactive focuses on user-facing AI experiences by measuring token generation performance under stringent latency requirements on metrics such as time-to-first token and token-to-token latency. Together, these scenarios provide a comprehensive view of both throughput and real-time serving performance.
Results Summary
| Scenario Throughput (Tokens/second) | |||
| Platform | Interactive | Offline | Server |
| ND GB300 NVL72 (18 nodes) | 260,098 | 661,143 | 589,391 |
| ND GB200 NVL72 (18 nodes) | – | 542,075 | 426,796 |
| ND GB200 NVL72 (72 nodes) | – | 2,091,190 | 1,598,510 |
Beyond the top results on GB200, Azure’s GB300 submission achieved 97.7% of the best reported Server performance and 95.8% of the best reported Offline performance while maintaining the highest Interactive performance among all reported submissions. Together, these results demonstrate competitive performance across diverse deployment scenarios and highlight the strength of Azure’s full-stack AI infrastructure.
Acknowledgments
This submission was led by Mark Gitau and Azin Heidarshenas, who drove the benchmarking, performance characterization, and submission efforts across GB200 and GB300 platforms.
MLPerf from MLCommons®
MLCommons® is an open engineering consortium of AI leaders from academia, research, and industry where the mission is to “build fair and useful benchmarks” that provide unbiased evaluations of training and inference performance for hardware, software, and services—all conducted under predetermined conditions.