NVIDIA Dynamo on AKS – Autoscaling LLM Inference

Large Language Models (LLMs) are rapidly moving from experimentation to production. While training often gets the spotlight, inference at scale is where real-world complexity emerges—especially when performance, cost, and user experience must all be balanced under unpredictable traffic. 

In this post, we’ll walk through how Azure Kubernetes Service (AKS) combined with NVIDIA Dynamo enables scalable, production‑ready LLM inference, and how to think about autoscaling GPUs effectively for modern AI workloads.

Why LLM Inference Autoscaling Is Hard

1. Variable and Non-Linear Metrics

Each request can vary significantly in compute cost depending on:

• • Model size and precision
  • Input and output token length
  • KV cache hit rate

GPU or memory utilization alone is not enough to represent real load.

2. Multi-Phase Inference Pipelines

LLM inference is not a single step. It consists of:

• • Prefill (context processing)
  • Routing
  • Decode (token generation)

Each phase has distinct performance characteristics and resource requirements, making coordinated scaling essential.

3. GPU Autoscaling Constraints

GPUs are:

• • Slower to provision than CPUs
  • Less elastic
  • Less available & more expensive 

Efficient packing, warm capacity, and informed scaling decisions are critical.

Choosing the Right Azure GPU Infrastructure

Azure offers a broad AI‑optimized VM portfolio designed for inference and training workloads. Selecting the right GPU SKU depends on:

• • Model size
  • Throughput requirements
  • Latency SLOs
  • Cost targets

AKS allows these GPUs to be managed through autoscaling node pools, forming the foundation for elastic inference platforms.

NVIDIA Dynamo on AKS: The Core Building Blocks

NVIDIA Dynamo is designed specifically for production LLM workload. It has several key capabilities:

• • Smart Routing – Routes requests efficiently across workers
  • KV Cache Management – Improves reuse and memory efficiency
  • Low-Latency Communication – Optimized GPU‑to‑GPU and service communication
  • GPU Planner – Coordinates capacity decisions across the pipeline

Together, these components enable Dynamo to treat inference as a system, not just a set of pods.

The “Autoscaling Magic Sauce”: Dynamo + AKS

What makes this architecture powerful is how Dynamo integrates natively with Kubernetes and Azure:

– Dynamo Scaling Adapters

Dynamo exposes Kubernetes scale subresources, allowing it to work seamlessly with:

• • Kubernetes HPA
  • KEDA
  • Dynamo Planner
  • Custom controllers

– AKS GPU Node Pool Autoscaling

AKS automatically scales GPU node pools based on pending pod demand using the cluster autoscaler.

– Observability Built In

By integrating with Azure Managed Prometheus and Grafana, you get:

• • Real-time inference metrics
  • Visibility into latency, throughput, and queue depth
  • Data-driven scaling decisions

Four Ways to Autoscale Dynamo Workloads

Dynamo supports multiple autoscaling strategies:

1. Kubernetes HPA

Best suited for:

• • Simple CPU or memory‑based scaling
  • Frontend or lightweight services

2. KEDA (Event-Driven Autoscaling)

Ideal when:

• • External or custom metrics are required
  • Scale‑to‑zero
  • Preset fixed max number of replicas

3. Dynamo Planner (LLM‑Aware Autoscaling)

Designed specifically for production LLM inference:

• • Adaptive scaling – no preset max number of replicas
  • Disaggregated scaling of prefill and decode workers
  • Uses live metrics and profiling data
  • Supports SLA‑driven goals (e.g., TTFT < 500ms)

4. Custom Controllers

Used for advanced or bespoke scaling logic when standard options are insufficient.

Metric Alignment: Scaling the Right Way

A key lesson in production inference is metric alignment—each service type should scale on signals that actually reflect pressure:

• • Frontend Services
    • CPU utilization
    • Request rate
  • Prefill Workers
    • Queue depth
    • Time to First Token (TTFT)
  • Decode Workers
    • KV cache utilization
    • Inter-token latency (ITL)

When metrics match service behavior, scaling becomes both faster and more stable.

Avoiding Common Autoscaling Pitfalls

A few best practices can save weeks of troubleshooting:

• • Use more than one autoscaler per service
    Avoid multiple autoscalers fighting over the same workload.
  • mis-configure stabilization windows
    Prevent rapid scale‑down that causes thrashing.
  • Arbitrary min/max replicas
    Protect the cluster from unbounded scaling or accidental scale‑to‑zero.
  • Understand your traffic patterns
    Short, spiky workloads require different strategies than steady-state serving.

Example: TTFT‑Driven Autoscaling with KEDA (Aggregated Serving)

This example shows NVIDIA Dynamo aggregated serving on AKS, where Prefill and Decode run in the same GPU worker pod. Autoscaling is handled by KEDA, using TTFT p95 latency from Prometheus as the scaling signal instead of CPU or GPU utilization.

Configuration summary

• Serving mode: Aggregated (Prefill + Decode per pod)
• Autoscaler: KEDA
• Scaling signal: TTFT p95 > 300 ms
• Worker replicas: 2 → 4
• Scale‑down cooldown: 120 seconds
• GPU: NC‑H100

What this load represents

• Model: Qwen3‑0.6B, small but latency‑sensitive under long prompts
• Traffic pattern: Constant request rate (36 RPS) with no traffic drop‑off
• Context pressure: Large input size (≈3000 tokens) to stress Prefill
• Streaming enabled: TTFT becomes the dominant user‑perceived metric
• Duration: Long‑running test to expose cold vs warm scale‑up behavior

What the chart shows

The chart below plots TTFT p95 over time while traffic remains constant.

Event Timeline

Warm vs. cold scale‑ups
The second and third scale‑ups are fast only because AKS has not yet terminated the extra GPU nodes. If the nodes had been removed, the next scale‑up would again incur the ~10‑minute cold‑start penalty.

Notebook & Code:
Dynamo_on_AKS/autoscaling_agg_keda/dynamo-keda.ipynb at main · maljazaery/Dynamo_on_AKS

How can I implement this? 

Step by step notebooks available here:
maljazaery/Dynamo_on_AKS: End-to-end tutorials for running NVIDIA Dynamo on Azure Kubernetes Service (AKS)