Delta Weight Sync in OpenRL

In RL post-training, the inference engine generating rollouts needs the latest policy weights after every optimizer step. OpenRL initially supported LoRA post-training, which updates only a set of adapters and makes syncing weights between the trainer and sampler lightweight. When we added full fine-tuning (FFT) support, transferring the full set of model weights after every optimizer step became one of our largest bottlenecks. This post explains how we solved that problem by syncing only the weights that actually change, as exact in-place patches applied directly by vLLM.

The weight sync bottleneck

OpenRL runs the trainer and the sampler as separate workers across GPU nodes, exchanging weights after every step through a shared Filestore (NFS). While NFS is convenient to mount across pods, it isn’t ideal for high-throughput checkpoint I/O. That didn’t matter for LoRA, where each sync moves only a few hundred megabytes of adapters, but with full fine-tuning on Gemma 4 E4B, syncing the full 16 GB checkpoint ^[3] over Filestore takes over two and a half minutes per step (roughly 80% of total step time).

Fortunately, almost all of that transfer is redundant. Recent work shows that RL updates are naturally sparse, changing only 5–30% of parameters across a full training run ^[1] and roughly 1% from one step to the next ^[2]. Our bf16 training runs show similar numbers (~0.7–2.5% of weights changing per step once training settles, depending on the learning rate). Instead of rewriting the full model every step, OpenRL uses delta checkpointing to sync only the weights that changed.

How delta sync works

1. Diff. The trainer keeps a copy of the weights it last sent to the sampler in pinned host CPU memory, requiring no extra GPU VRAM. After each optimizer step, it compares the new weights against that copy bit for bit and collects the elements that changed.
2. Write. It writes a delta to the shared filesystem with two pieces: which positions changed, and their new values, alongside metadata recording the modified layers and the share of weights that changed.
3. Apply. When the sampler's next request needs the new version, it pauses generation, applies the delta in place, and resumes. It doesn't restart or reload the model.

The delta format. Each delta is a standard safetensors file (delta.safetensors) structured directly for vLLM’s checkpoint_weight_patch loader: it stores a list of changed indices and their new weights for each modified layer, coming out to 6 bytes per changed weight at half-precision (bf16). Unchanged tensors are omitted, so any step with <33.3% mutation density is smaller than a full checkpoint dump.

• Because each patch stores the new parameter value rather than an additive difference, applying it reproduces the trainer’s weights bit for bit, making updates idempotent and free of rounding drift over thousands of steps.
• A small metadata.json alongside each delta records:
  • the names and shapes of the modified weight tensors (layer_names, layer_shapes)
  • how many weights changed (changed_elements, total_elements)
  • the change density (density_pct) and format version (sparse_delta v2)

Applying weights quickly with vLLM

On the sampler side, applying a delta means updating the changed parameters in place inside a running vLLM engine without restarting it or dropping the KV cache. We build on two vLLM primitives:

1. Custom weight transfer backend.

• vLLM provides a pluggable weight transfer engine ^[4] interface that receives weight updates while generation is paused, driven through start, update, and finish hooks.
• OpenRL registers a delta_snapshot backend enabled via WeightTransferConfig(backend="delta_snapshot"), avoiding a custom vLLM fork.

2. In-place sparse patching.

• Inside that backend, we pass the changed positions and values to vLLM’s checkpoint weight patch loader, indexed by the original Hugging Face parameter names.
• Even though vLLM fuses layers (such as Q, K, and V projections) and shards tensors across GPUs, the patch loader maps the Hugging Face indices to vLLM’s internal serving layout and writes the updates directly into GPU memory.

Because vLLM handles the layout mapping, the same delta format works across Gemma 4 and Qwen3 without model-specific code in OpenRL. Thank you to the vLLM team for making weight transfer pluggable!

Results: GRPO on GSM8K

We evaluated full-checkpoint sync against delta sync during full-parameter GRPO training on GSM8K using OpenRL’s PyTorch trainer (8 groups of 8 rollouts per step, 512 max tokens, 100 steps, 1×H100 trainer node and 1×H100 sampler node connected via the Zonal Filestore NFS share, averaged across 3 seeds).

Because each delta transfers only a small fraction of the full checkpoint, both the trainer’s NFS write and the sampler’s update pause drop from over a minute to just a few seconds. Weight sync goes from dominating each RL step to a minor overhead, speeding up end-to-end training by roughly 5–6× across both Gemma 4 E4B and Qwen3-8B-Base.

Mutation density over training

Every delta records its mutation density: the fraction of model weights that changed on that step. Because the trainer writes these deltas as part of weight sync, we get a step-by-step view of how the policy mutates alongside reward at no extra overhead.

Generalizing across open-weights models and prior literature

OpenRL also supports open-weights model families such as Qwen3 (Qwen3-8B-Base and Qwen3-4B-Base), which show the same sharp step-1 spike followed by low steady-state mutation density:

The Qwen3 curves sit higher than Gemma 4 E4B partly because they use a higher learning rate (1e-5 vs. 3e-6), and partly because Gemma 4 E4B’s 8B total parameters include multimodal vision/audio encoders and Per-Layer Embedding lookup tables ^[3] that remain inactive on text-only GSM8K batches. That highlights an added benefit of delta sync on multimodal models: rather than wasting bandwidth rewriting gigabytes of frozen or inactive modality parameters on every step, unchanged tensors are skipped automatically. Across all three models, 84–97% of weights never change over the full run, consistent with prior sparsity studies ^{[1, 2]}.

Future work

Acknowledgements

Thank you to Hoda Mohammadi and Steve Linde for their thoughtful reviews and feedback on this post.

References

1. S. Mukherjee, L. Yuan, D. Hakkani-Tür, H. Peng. Reinforcement Learning Finetunes Small Subnetworks in Large Language Models. arXiv:2505.11711, 2025.
2. E. Miahi, E. Belilovsky. Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL. arXiv:2602.03839, 2026.
3. Gemma Team. Gemma 4 Technical Report. arXiv:2607.02770, 2026 (Table 1: Gemma 4 E4B has 4.5B effective text parameters and 7.94B total parameters including 2.82B Per-Layer Embedding lookup parameters and multimodal encoders, totaling 16.0 GB in bf16).
4. vLLM. Weight Transfer documentation.
5. P. Stefanopoulos. Optimizing delta weight syncs for managed RL rollouts. Baseten Blog, September 2026.