A practical guide to running 8x RTX PRO 6000's

When we got our hands on an 8x NVIDIA RTX PRO 6000 Blackwell system paired with an AMD EPYC 9555 (64 cores, 128 threads, Zen 5), the first question we asked was simple: what can we run with this?

With 768 GB of aggregate GDDR7 VRAM on tap, the knee-jerk instinct was to test whether we could shard larger models like Llama-3.1 405B or DeepSeek-R1 671B across all eight cards. But that seems a job more suited to this config's much more expensive sibling in NVIDIA's data center lineup.

That realization led us to reframe the architecture.

Instead of forcing deep model parallelism over PCIe, it makes more sense to explore what workloads we can handle at a fraction of the price of something like an HGX B200/B300 without sacrificing too much performance.

1. Interconnect Constraints: PCIe Gen 5 vs. NVLink

Interconnect bandwidth and collective latency determine whether a workload scales efficiently via model parallelism or independent concurrency:

Although going with PCIe 5.0 clearly has limitations, the advantages and capabilities of NVLink you see above also come attached to an abnormally high price tag.

Now, this is unavoidable when you're training or serving frontier models that span 100s of gigabytes or even terabytes. But you can handle concurrent agentic workloads and so much more at a fraction of the price with Blackwell GPUs attached to the good old PCIe slot.

Tensor parallelism in motion: one model sharded across GPUs and stitched back together over the interconnect. Animation: NVIDIA.

2. Recommended Parallelism Strategies

So, if you aren't sharding massive 400B+ monolithic models across this server, what CAN it do? Well, servicing very high concurrency needs, of course, and some more:

A. Isolated Single-GPU Serving (TP=1, DP=8)

The 96 GB GDDR7 capacity of each RTX PRO 6000 accommodates dense mid-sized architectures like the new Qwen3.8-27B entirely on a single card without multi-GPU sharding. What this means is that your mileage will vary depending on your model and the precision used:

B. Multi-Model Microservice Fleets (8B-32B)

For enterprise inference routing, each 96 GB card can host multiple smaller models (e.g., Llama-3.1 8B, Mistral Small, Qwen3 32B) or deep continuous-batching workers. With this kind of strategy, the numbers break down to something like this:

C. Shard Fine-Tuning with Overlapped Communication

During parameter tuning, frameworks like PyTorch FSDP and DeepSpeed ZeRO-3 overlap PCIe gradient all-gathers and reduce-scatters with backward-pass compute. Sharding states across the 768 GB aggregate pool supports fine-tuning 70B+ architectures. However, full 70B AdamW fine-tuning necessitates mandatory CPU offloading (~840 GB optimizer states requiring at least 1.15 TB host RAM via 12x 96GB DIMMs). At 8 GPUs transferring 64 GB/s each, the aggregate 512 GB/s PCIe demand operates at ~83-89% of the EPYC 9555's 576-614 GB/s DDR5 bus, making host memory bandwidth co-limiting alongside PCIe.

3. Orchestration Using the AMD EPYC 9555

While the unbelievably powerful RTX 6000 Blackwell GPUs handle raw matrix multiplication and token generation, feeding eight independent high-throughput accelerators requires significant host-level coordination. In an asynchronous, multi-tenant system, that's where the CPU comes into the picture.

Here's how the 64-core EPYC 9555 and its 12-channel memory subsystem can keep the accelerator pool fully saturated without becoming an operational bottleneck:

• Host RAM for staging, not active KV paging: The EPYC 9555's 12-channel DDR5 bus achieves up to 614 GB/s theoretical peak host memory bandwidth (or ~576 GB/s at DDR5-6000). While paging dynamic KV caches across the 64 GB/s PCIe boundary during live token generation causes severe latency spikes, host memory excels at prefix-cache staging, asynchronous RAG vector buffering, and FSDP optimizer offloading.
• High-throughput pre-processing: Managing 8 to 32 parallel inference and diffusion streams requires quite a bit of CPU compute. The 64 Zen 5 cores handle tokenization, JSON grammar enforcement, embeddings, and network routing in parallel without thread contention.

4. Production Workloads & Throughput

To translate these architectural constraints into practical capacity planning, the table below maps real-world workloads to their optimal deployment models on this 8-GPU platform. It highlights how throughput, latency, and operational efficiency shift across isolated single-GPU instances (TP=1), paired dual-GPU serving (TP=2), and asynchronous batch pipelines, showing where eliminating inter-GPU PCIe traffic gives you the best return in tokens.

Summary

This platform isn't the 'best' for massive frontier-model experiments; that's a job for the cutting-edge in speed and memory bandwidth from NVIDIA's stable, which of course, do cost quite a bit more. Where this platform shines is concurrency. And you can get yourself an excellent bargain in compute when you run the right workloads on this type of architecture. You know, things like:

• Stall-free serving: Run eight separate 27B or 70B models at full speed, eliminating inter-card PCIe synchronization bubbles and maximizing memory bandwidth saturation.
• Deep concurrency: Host thousands of simultaneous chat sessions and agent reasoning loops with massive dedicated KV cache headroom.
• Microservice fleets: Co-locate dozens of specialized 8B-32B coding, vision, and tool-calling models on a single machine.
• Local fine-tuning: Train 70B+ models natively using your host RAM and 768 GB VRAM pool without paying cloud multi-node premiums.

Availability

You can find out more about these systems on our product pages: the CG480-S6053 4U GPU server this build lives in, and the RTX PRO 6000 Blackwell cards themselves.

We currently have 14x nodes of CG480-S6053 ready to ship. No lead time. The full L10 spec sheet (PDF) covers the exact build.

Sources