OpenRL

GPUs for work, not for people.

A post-training service you host yourself. RL and SFT on your Kubernetes cluster, behind the Tinker API.

OpenRL Authors

github.com/openrl-dev/openrl

Agenda.

  1. 1Post-training and where RL fits
  2. 2Three problems with post-training today
  3. 3OpenRL: scheduling work units, not GPUs
  4. 4How OpenRL is built
  5. 5Benchmarks and resource utilization
  6. 6Getting started and roadmap

Part 1

Post-training turns a base model into a specialist.

Part 3

OpenRL is a post-training service for Kubernetes.

Part 4

Architecture and execution model.

Part 5

Benchmarks and resource utilization.

Part 6

Getting started and roadmap.

Part 2

Three problems with post-training today.

OpenRL is a post-training service you host yourself.

It runs RL and SFT post-training on your Kubernetes cluster, behind the Tinker API.

Self-hosted
Deploys on your Kubernetes cluster. Data and weights stay in your network.
Tinker-compatible
Implements the Tinker API. Existing Tinker scripts and recipes run unchanged.
Shared GPU pool
Interleaves LoRA and full fine-tuning jobs across shared GPUs.
Multi-GPU training
Scales trainers up to 8 GPUs with FSDP2 and context parallelism; scales samplers as replicas.
Custom trainers
Supports custom trainer container images, frameworks, and accelerator stacks.
Programmatic API
Exposes HTTP endpoints for model status, job settings, cluster health, and coding agents.

Small fine-tuned models beat frontier models.

The bottleneck has shifted from model architecture to running the RL loop.

Post by tobi lutke (@tobi), 8:24 AM, Sep 1, 2026: Training tiny models for special purpose use cases works so incredibly well if you have a great self improving recursive flywheel. Shopify ML team is on fire. finetuned 0.8b model beats GPT 5.6-sol xhigh in this very specialized task. Attached: a Buyer Profile case-study chart, judge score 83.0 to 84.6, system prompt 9.1K to 1.1K tokens, profiles per day 2M to 72M. 8.5K likes, 312 replies.
x.com/tobi/status/2094808564355191249
Post by Jasper Lu (@lu__jasper), 11:44 AM, Sep 1, 2026: It took a few years, but I think we’re finally nearing an xgboost moment for llm training. When xgboost came out, it made training models on tabular data so easy that every company could suddenly ship fast and useful models without a real research team. 450 likes, 8 replies.
x.com/lu__jasper/status/2094858986419843371

The hard part now is running the loop: GPUs, scheduling, weight sync, and job lifecycles.

RL is a loop.

Sample from the current policy, grade the samples, update the weights, hand them to the sampler. Repeat for a few hundred steps.

Sample generate rollouts sampler GPU, inference Grade score each rollout CPU: tools, sandboxes, judges Train forward_backward, optim_step trainer GPU Sync weights trainer to sampler every step, to stay on-policy repeat Infrastructure beneath every phase bundled into the training script today inference servers GPU memory and placement loading each step's weights sandboxes and tool servers judge endpoints retries, timeouts, rate limits trainer processes, CUDA GPU placement and scheduling checkpoints and resume storage and transport which weights the sampler holds recovery when a worker dies Each phase waits for the one before it, so a job on its own leaves its GPUs idle most of the time. Today all of this sits in the researcher's code. That coupling is the problem.

The loop is user code.

Write plain Python with the Tinker SDK. The environment, reward, and loss function stay in your code.

import tinker
svc = tinker.ServiceClient(base_url="http://openrl.my-cluster:8000")
trainer = await svc.create_lora_training_client_async(
    base_model="Qwen/Qwen3-4B-Instruct-2507", rank=16)

for step in range(40):
    sampler = trainer.save_weights_and_get_sampling_client(name=f"step-{step}")
    rollouts = sampler.sample(prompt, num_samples=8, sampling_params=params).result()
    datums = score(rollouts)                      # your env, your reward
    await trainer.forward_backward_async(datums, loss_fn="importance_sampling")
    await trainer.optim_step_async(tinker.types.AdamParams(learning_rate=1e-5))
save_weights
Publish the trainer’s weights so the sampler sees them.
sample
Rollouts from the new weights, in parallel per prompt.
forward_backward
First half of a training step. You choose the loss.
optim_step
Second half. You choose the optimizer.

OpenRL separates control plane from data plane.

The control plane queues API requests and schedules workers. The data plane runs training and sampling jobs.

Kubernetes cluster OpenRL control plane OpenRL data plane RL loop Tinker SDK runs on laptops or in-cluster Pods API server Tinker-compatible API over HTTP forward_backward, optim_step, save_weights, sample, plus checkpoints worker CRs Scheduler Go controller for worker lifecycle DRA sizes each worker's GPU allocation from model and batch size creates and places workers enqueues each request Queue a queue per job, Redis pull workers: shared across LoRA jobs, or time-sliced per GPU Trainer workers forward_backward, optim_step Sampler workers sample · vLLM, 1 to 8 replicas writes weights each step reads them Shared filesystem checkpoints, and the weights that move from trainer to sampler every step

Workers share GPUs by adapters or by time-slicing.

LoRA jobs on the same base model share a running vLLM and trainer process. Full fine-tuning jobs time-slice the GPU and offload inactive state to host memory.

Six RL runs train in parallel on one shared GPU pool.

Reproducing LoRA Without Regret on GSM8K RL: full fine-tuning against LoRA rank 32 and rank 1, two seeds each, six runs training at once on one OpenRL cluster.

Mean reward, last 10 of 40 steps

full fine-tuning
0.934
LoRA rank 32
0.957
LoRA rank 1
0.936

Seed-to-seed variance is 0.013 to 0.016; LoRA rank 1 matches full fine-tuning within noise.

All six runs trained concurrently. Four LoRA runs shared one trainer and one sampler process, while two full fine-tuning runs time-sliced a single GPU allocation.

Qwen3-8B on H100, 40 steps of 64 episodes. Line: mean of two seeds, 5-step rolling mean. Band: the spread between the seeds. Data: docs/experiments/lora-without-regret on branch experiment/lora-without-regret. Paper: thinkingmachines.ai/blog/lora

Time-slicing more than doubles trainer utilization.

Running three jobs on two shared H100s instead of six dedicated H100s yields identical learning curves at 2.2x the trainer duty cycle.

Duty cycle over time for six dedicated baseline GPUs, each mostly idle, against two time-sliced GPUs that stay busy across all three jobs

Time-sliced, three jobs on two H100s

Trainer duty cycle
34.2%
GPUs provisioned
2
GPU-hours
1.30
Wall clock, all three
39 min

On six dedicated GPUs: 15.6% duty cycle and 2.10 GPU-hours. The same two GPUs running the jobs one after another: 54 minutes.

Context switches take 0.5 to 1.6 s (median), and phase queue waits take 1 to 15 s. All three jobs matched their dedicated-GPU learning curves.

Jobs: Text-to-SQL RL on Qwen3-1.7B, math RL on Qwen2.5-7B, dialogue SFT on Gemma 4 E2B. Figure from llm-d.ai/blog/increase-researcher-velocity-rl-llm-d-time-slicing

OpenRL installs with one manifest.

Create a cluster with shared storage, install the NVIDIA and DRA drivers, apply the manifest.

Install

1

Create a GKE cluster on 1.35 or newer, with Filestore as the shared filesystem.

$ gcloud container clusters create ml \
    --location us-central1 --release-channel regular \
    --addons GcpFilestoreCsiDriver
2

Add a GPU pool that opts in to the scheduler. Install the NVIDIA driver (GKE’s installer DaemonSet) and the NVIDIA DRA driver.

$ gcloud container node-pools create gpus --cluster ml \
    --accelerator type=nvidia-l4,count=2,gpu-driver-version=disabled \
    --node-labels openrl.io/enabled=true,\
                  gke-no-default-nvidia-gpu-device-plugin=true
$ kubectl apply -f nvidia-driver-installer/cos/daemonset-preloaded-latest.yaml
$ helm install nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu
3

Install OpenRL. The standard manifest enables both LoRA and full fine-tuning.

$ kubectl apply --server-side -f https://github.com/openrl-dev/openrl/releases/latest/download/openrl-fft.yaml

Use

Point the Tinker SDK at the cluster URL. The script from Part 3 runs unchanged, with OpenRL settings passed in user_metadata.

import tinker
# your cluster, not Tinker's cloud
svc = tinker.ServiceClient(
    base_url="http://openrl.my-cluster:8000",
    user_metadata={"openrl.fine_tuning_type": "full"})
# the SDK has one client type; the setting above makes it full fine-tuning
trainer = await svc.create_lora_training_client_async(
    base_model="google/gemma-4-e2b", rank=16)
# then sample, score, forward_backward, optim_step

openrl-doctor --base-url … --train checks the deployment end to end. The full walk-through is docs/getting-started.md.

Current capabilities and six-month roadmap.

What runs in OpenRL today, and what we are building next.

Production readiness

nowRuns our internal workloads and the Harvey legal benchmark (LAB). Single-manifest install; per-job configuration.

nextDay-0 and day-2 operating guides. A versioned API with an upgrade path.

Multi-GPU per host

nowAutomodel trainers on up to 8 GPUs: FSDP2 and context parallelism. Sampler replicas.

nextMulti-GPU full fine-tuning and time-slicing, tensor-parallel samplers, Megatron.

More accelerators

nowNVIDIA L4 and H100 through Kubernetes DRA.

nextA TPU proof of concept, then an MVP.

More models

nowDense Qwen and Gemma to 10B on one GPU; Gemma 4 31B with multi-GPU LoRA trainers.

nextFull fine-tuning at 30B. MoE.

Observability

nowModel status API, capabilities and settings schema over HTTP, openrl-doctor, an agent skill.

nextA CLI agents can consume, a dashboard, per-step metrics, cancelling a job.

Recipes

nowHarvey LAB (agentic, sandboxed), Text-to-SQL, math RL, SFT notebooks, autoresearch.

nextFinancial domain recipes, paper reproductions, custom sampler images, and the checkpoints API.

The full plan, by focus area and initiative, is in ROADMAP.md. To change it, open an issue with the roadmap label.

Build it with us.

We have applied to donate OpenRL to the CNCF, for a vendor-neutral home.

github.com/openrl-dev/openrl

The roadmap and the issues are public. File one, or pick one up.

Try it

  • Getting-started guide: go from a fresh GPU cluster to your first SFT and RL run.
  • Harvey LAB recipe: run sandboxed agentic RL as a Kubernetes Job.
  • Text-to-SQL RL recipe: run a complete sample, grade, and train loop.
  • Coding agent skill: install the skill so your coding agent can submit and monitor RL runs.

docs/getting-started.md · examples/harvey_labs · examples/text-to-sql · openrl-skill.tar.gz on every release

Thank you.

GPUs for work, not for people. Standard RL loops on shared Kubernetes accelerators.

OpenRL Authors

github.com/openrl-dev/openrl

Three problems OpenRL is built to solve.

All three come from the same setup: one job, one stack, one block of GPUs.

1. GPUs are reserved by people
A job holds its GPUs for the length of a reservation and idles between phases. Nobody gives them back, because getting them again takes weeks.
2. Research and infrastructure ship together
The training script carries the framework, vLLM, NCCL, weight sync and checkpoints. The researcher maintains all of it, and the platform team cannot help.
3. Agents multiply the runs
A coding agent wants twenty variants tonight. One job at a time, on one reservation, cannot keep up.

Syncing weight deltas makes each RL step 6x faster.

The trainer diffs current weights against the previous step and writes only the changed elements. The sampler patches weights in place in vLLM without restarting or reloading 16 GB.

Median time per RL step split into trainer write, sampler pause, and rollouts plus training, full checkpoint against delta sync, for Gemma 4 E4B and Qwen3-8B-Base

Gemma 4 E4B, per step

Bytes synced
16 GB → 0.38 GB
Trainer write
68 s → 2.2 s
Sampler pause
91 s → 2.4 s
RL step
191 s → 32 s

Qwen3-8B-Base sync volume drops from 16.4 GB to 1.48 GB, and step time drops from 189 s to 36 s.

A delta is a standard safetensors file for vLLM's checkpoint weight patch loader, so one format serves Gemma 4 and Qwen3 with no model-specific code.

GRPO on GSM8K, 8 groups of 8 rollouts, 100 steps, one H100 trainer and one H100 sampler over Zonal Filestore NFS. Medians per step, mean of 3 seeds. Figure from openrl.dev/blog/delta-weight-sync.

Under 1% of Gemma 4 E4B's weights change per step.

3.3% of weights change on step 1, then fewer than 1% on later steps, while GSM8K accuracy climbs from 39% to 67%.

Per-step mutation density and GSM8K training accuracy for Gemma 4 E4B over 100 GRPO steps

Gemma 4 E4B, lr 3e-6

Step 1
3.3%
Steps 10 to 20
0.90%
Last 10 steps
0.66%
Never changed in 100 steps, across three models
84 to 97%

Each delta file records which layers changed and by how much, giving per-step layer mutation metrics at no extra cost.

Qwen3-8B-Base and Qwen3-4B-Base show the same shape at a higher learning rate.

Problem 1 of 3One researcher, eight GPUs, one week.

Nobody releases idle GPUs. When re-queuing takes two weeks, hoarding is the rational choice.

Problem 2 of 3The researcher ends up owning the stack.

Research and infrastructure ship as one container, so one person maintains both.

Problem 3 of 3Coding agents multiply the number of parallel runs.

researcher
+ coding agent
reward v1 … v20, same base model, same data
8 GPUs, one job at a time
160 GPUs?

RL-as-a-service used to be built for an entire org. With coding agents, a single researcher generates enough jobs to need one.

Kubernetes schedules Pods; OpenRL schedules RL work units.

Kubernetes schedules Pods, not servers
three teams' services web search billing become Pods placed on any node
OpenRL schedules sampling and training units, not GPUs
your script, Tinker SDK for step in range(N): sample → reward → forward_backward → optim_step → save_weights no GPU named anywhere Tinker API · every call goes to a service OpenRL backend: the stream of calls from every loop becomes work units one pool, units from three jobs interleaved

The training loop stays in your script. OpenRL turns API calls into sampling and training units and places them across a shared GPU pool.

Pool scheduling replaces per-job GPU reservations.

OpenRL integrates standard Kubernetes and ML tools.

OpenRL provides the API server and the work-unit scheduler, built on standard open-source training and serving runtimes.

Extras

Spare slides.

Decouple infra from AI research.

The researcher's code stays on top, OpenRL runs underneath, and the Tinker API is the contract between them.

AI researcher plain Python, on a laptop or in the cluster Dataset and environment rewards, tools, sandboxes RL algorithm GRPO, PPO, loss, advantages The loop itself tinker-cookbook, or hand-written Same loop. Any model, any accelerator. Tinker API: forward_backward, optim_step, save_weights, sample OpenRL self-hosted on your Kubernetes cluster API server, queue many jobs at once Trainers LoRA and full fine-tuning Samplers vLLM, delta weight sync Scheduler DRA, time-slicing GPUs shared across jobs; researchers never see them

Many RL loops, one pool.

Each job is a researcher’s own loop. They start together, and OpenRL places their workers on whatever GPUs the pool has.

How a GPU is shared.

Full fine-tuning needs the whole device, so co-located workers take turns. A timeslicer grants the turn; llm-d's snapshot agent parks the state in between.

Kubernetes node joins the trainer or sampler pool with one label; workers land on it as pods Trainer, tenant A pod: forward_backward, optim_step Sampler, tenant A pod: sample Trainer, tenant B pod: forward_backward, optim_step acquire, yield Timeslicer node-local DaemonSet, one holder at a time workers reach it over a local socket 1 a worker asks to acquire 2 the holder yields; its state goes to DRAM 3 the next state returns to HBM; it runs llm-d snapshot agent parks accelerator state in DRAM and restores it for the next holder the holder computes Accelerator who holds it, over time A trains B trains A samples A trains B trains A samples snapshot restore Host DRAM parked state of the others A parked worker resumes from the same state when its turn comes back.

Delta weight sync: faster weight transfer.

Qwen3-8B on GSM8K math RL. The share of parameters that change in a step falls from 12% at step 1 to under 3% by step 25, while reward climbs. So the trainer ships only the changed parameters, and the sampler patches them in place.

Model weights mutation versus reward progression for Qwen3-8B on GSM8K math RL over 49 steps: the mutated share falls from 12.9% at step 1 to about 2.6% by step 25 and stays near 3%; reward rises from 0.33 to about 1.0 by step 10

Delta weight sync, Qwen3-8B

Weights changed, step 1
12%
Weights changed, step 50
2.6%
Bytes per sync
1.2 GB
Sampler weight load
5 s

A full checkpoint is 16 GB and took 35 s to load on the sampler. Deltas travel over shared NFS on GKE, so trainer and sampler need no RDMA link and can be placed independently.

The trainer already holds the previous weights in host memory for time-slicing; that is what the delta is computed against.

Figure from the note “Delta Weights in RL” (Jul 2026). Per-step density for the same model is in docs/experiments/lora-without-regret/delta_density.csv on branch experiment/lora-without-regret.

Once jobs are broken into work units, four things follow.

Share

LoRA jobs on one base model share a trainer and a sampler. Full fine-tuning jobs take turns on a GPU through a time-slicer built with llm-d.

Right-size

The batch and the model set what a unit needs. Dynamic Resource Allocation carves an accelerator shape to fit, so the pool packs tighter.

Sync only the delta

After the first few steps, under 1% of Gemma 4 E4B's weights change per step. Send only those: 0.38 GB instead of 16 GB, and an RL step drops from 191 s to 32 s.

Queue, do not reserve

A job waits for its next unit, not for eight GPUs. A new job starts as soon as a unit fits, and a finished job frees the pool at once. Nothing is held idle.

A sweep an agent could launch tonight.

24 concurrent GRPO jobs on six H100s

LoRA and full fine-tuning mixed. Every job finished. The whole batch took eight minutes.

0.57 → 0.72 held-out reward, 30 steps

Harvey's Legal Agent Benchmark: documents in a sandbox, tools to read them, deliverables graded by a judge. Recipe in the repo, 32k-token turns in a 192k context.

openrl.devgithub.com/openrl-dev/openrlCNCF Sandbox projectAgent skill in every release

Pre-training gives you the base model.

Every capability we care about is already in it, at low probability.

Post-training (RL) sharpens what is already there.

RL raises the probability of what you reward.