The change is described for DeepSeek-V4.1-Flash running data-parallel attention with expert-parallel MegaMoE across multiple NVLink-connected nodes.
The PR reports tests on a GB300 NVL72 system using DSpark speculative decoding with five draft tokens and coding-agent prompts averaging about 85,000 prefix-cached input tokens. At DEP32 and 128 requests per rank, it reports throughput rising from 710,600 to 897,100 tokens per second, or 26.2%. It also reports main-model MegaMoE time falling from 14.65 ms to 7.62 ms per decode step.
The proposed backend uses CUDA symmetric memory for expert-row transfers; the author reports cross-node peer copies of about 800 GB/s and a 40-layer rearrangement in 0.3–0.6 seconds at DEP32. These are PR-reported measurements in a specific hardware, model, workload, and nightly-build configuration. The source does not establish that the change has merged into vLLM, shipped in a stable release, or produces the same gains on other systems.