AI Infrastructure Preflight at User space: Validating Multi Node, Multi GPU Slurm Clusters

Every team that operates GPU clusters for AI has seen this pattern. The cluster boots, GPUs are visible, and scheduling works at a basic level. Then the first distributed training run stalls in NCCL initialization, fails during rank rendezvous, or silently maps ranks to the wrong devices. The issue is often not in training code. It is in infrastructure consistency across scheduler, runtime, drivers, networking, and process topology.

The goal of ai-infra-validator is straightforward:

• Run a fast user space preflight before expensive training jobs.
• Validate distributed initialization for multi node, multi GPU workloads.
• Confirm GPU affinity and rank mapping are correct.
• Verify NCCL communication fabric can complete a collective ring under Slurm.

This post walks through the implementation in detail, explains why each part exists, and shows how to operationalize it in real HPC AI environments.

What the project validates

Zero-dependency user space smoke test for AI clusters. Validates multi-node PyTorch DDP initialization, GPU affinity, and NCCL fabric connectivity under Slurm orchestration.

Git Repo: ai-cluster-validator

In practical terms, this checks that:

1. Slurm launches the expected number of ranks per node.
2. Distributed process group creation with NCCL succeeds.
3. Each rank binds to the expected local GPU.
4. Cross-rank all-reduce completes and converges.
5. Node level telemetry confirms software and fabric state.

This is not a performance benchmark. It is a correctness and readiness gate.

Tested platform profile

This profile represents a common enterprise scenario where scheduler and compute nodes have different roles, and the training fleet depends on correct multi node orchestration.

Step 1: Minimal user space bootstrap

The bootstrap script creates a shared Python environment at /shared/apps/pytorch_env and installs the required packages:

• torch
• torchvision
• torchaudio
• psutil

This choice is intentional:

• No dependency on containers for first-pass validation.
• Single environment path visible to all compute nodes.
• Rapid setup and repeatability for cluster operators.

Command sequence:

Step 2: Slurm job defines deterministic distributed topology

The Slurm script expresses a clear topology contract:

• nodes=2
• ntasks-per-node=8
• gpus-per-node=8
• cpus-per-task=12

From this, world size is derived as:

WORLD_SIZE = SLURM_NTASKS = 2 x 8 = 16

The script also configures network and NCCL behavior:

• NCCL_DEBUG=WARN
• NCCL_IB_DISABLE=0
• NCCL_P2P_DISABLE=0
• NCCL_IGNORE_CPU_AFFINITY=1
• GLOO_SOCKET_IFNAME=eth0
• NCCL_SOCKET_IFNAME=eth0

Important implementation detail:

• MASTER_ADDR is set to the first host in SLURM_JOB_NODELIST.
• MASTER_PORT is selected dynamically from the ephemeral range 49152-65535 and falls back to 29500 if needed.

Why this matters:

• Reduces port collision risk when jobs run frequently.
• Avoids hardcoded rendezvous values that may fail in shared clusters.

Launch path:

The LOCAL_RANK handoff is critical for stable GPU affinity inside each node.

Step 3: DDP initialization and rank to GPU affinity

Inside ddp_mesh_ping.py, each process executes:

1. Parse WORLD_SIZE, RANK, LOCAL_RANK, MASTER_ADDR, MASTER_PORT.
2. Initialize torch.distributed with backend nccl and TCP init method.
3. Set CUDA device using LOCAL_RANK.

Core initialization path:

This validates the minimum distributed contract required by real model training jobs.

Step 4: Rich node and fabric telemetry in user space

Each rank collects detailed metadata before the collective test:

• Node identity from Slurm and hostname.
• GPU model and VRAM from CUDA properties.
• System memory via psutil.
• CPU model from /proc/cpuinfo.
• OS and kernel versions.
• NVIDIA driver version from /proc/driver/nvidia/version.
• PyTorch, CUDA, and NCCL runtime versions.
• InfiniBand device state and link rate from /sys/class/infiniband.
• Basic GPU peer access capability via torch.cuda.can_device_access_peer.

All rank payloads are gathered on rank 0 using dist.gather_object and printed as:

• Cluster hardware topology report.
• Node environment deep dive.
• Network interconnect and fabric status.

This design gives platform teams one artifact that is both operational and diagnostic.

Step 5: Functional collective validation

After telemetry, each rank executes a lightweight DDP compute path:

1. Build nn.Linear(10,10) on local GPU.
2. Wrap with DistributedDataParallel.
3. Perform forward, loss, backward.
4. Run all_reduce on loss tensor.
5. Compute global average loss.

Pass condition is explicit in log output:

SUCCESS: DDP Multi-Node AllReduce Ring Complete!

This confirms that process group initialization and collective communication both completed successfully.

What a successful run looks like

Submission:

Representative outcomes in the log:

• Total Execution Ranks: 16
• Two nodes with local ranks 0 through 7 on each node
• GPU inventory aligned with expected A100 topology
• Active InfiniBand HCAs discovered per host
• NCCL socket interface set to eth0
• Final success marker and computed convergence loss

When these markers are present and coherent with expected hardware shape, the cluster is typically ready for distributed training bring-up.

How to check the output file

The Slurm script writes two artifacts per job:

• ai_infra_smoke_test_.log
• ai_infra_smoke_test_.err

Use this exact workflow after submission:

For stronger validation in automation, also check:

1. Total Execution Ranks equals expected world size.
2. Both nodes appear in the topology table with local ranks 0 through 7.
3. NCCL/CUDA/PyTorch versions are present in the node environment section.

Complete reference output

Use the following full log as a known-good reference from a successful 2-node ND96asr_v4 run.

Why this is effective for platform operations

For AI infrastructure teams, this pattern is highly effective because it is:

• Fast: can be run after every change window.
• Deterministic: same topology contracts every run.
• Actionable: output includes enough context for first-level triage.
• Low friction: user space only, no heavy control plane dependencies.

This supports common operating workflows:

1. Day-0 cluster acceptance.
2. Day-1 patch validation after driver, kernel, or image changes.
3. Regression gate in golden image pipelines.
4. Preflight before large multi node model training jobs.

Practical guidance for extending to larger clusters

1. Adjust Slurm directives for nodes and tasks per node.
2. Keep one rank per GPU unless validating alternate placement policy.
3. Set NCCL_SOCKET_IFNAME and GLOO_SOCKET_IFNAME according to your network policy.
4. Preserve the dynamic MASTER_PORT logic to avoid static collisions.
5. Keep the success marker string stable so automation can parse it.

Closing perspective

Most distributed training failures are expensive because they are discovered late. A user space preflight that validates scheduler topology, rank rendezvous, GPU affinity, and NCCL collectives provides a high value guardrail before production starts.

ai-infra-validator is a practical implementation of that guardrail. It is compact, transparent, and aligned with how real Slurm based AI clusters operate. For teams running multi node multi gpu training at scale, this kind of preflight should be a standard operational gate.