OpenRL

A self-hosted, Kubernetes-native post-training API for fine-tuning LLMs.

Sunil Arora
SWE@Google, droot@

github.com/gke-labs/open-rl

Agenda.

  1. 1Post-training, and where RL fits
  2. 2OpenRL: a self-hosted, Tinker-compatible post-training API
  3. 3How OpenRL is built
  4. 4Concurrent RL in action
  5. 5Getting started
  6. 6Roadmap and how to contribute

Part 1

What is post-training?

Part 2

What is OpenRL?

Part 3

How is OpenRL built?

Why now.

Small task-specific models are beating frontier models on narrow tasks.

Post by tobi lutke (@tobi), 8:24 AM, Sep 1, 2026: Training tiny models for special purpose use cases works so incredibly well if you have a great self improving recursive flywheel. Shopify ML team is on fire. finetuned 0.8b model beats GPT 5.6-sol xhigh in this very specialized task. Attached: a Buyer Profile case-study chart, judge score 83.0 to 84.6, system prompt 9.1K to 1.1K tokens, profiles per day 2M to 72M. 8.5K likes, 312 replies.
x.com/tobi/status/2094808564355191249
Post by Jasper Lu (@lu__jasper), 11:44 AM, Sep 1, 2026: It took a few years, but I think we’re finally nearing an xgboost moment for llm training. When xgboost came out, it made training models on tabular data so easy that every company could suddenly ship fast and useful models without a real research team. 450 likes, 8 replies.
x.com/lu__jasper/status/2094858986419843371

The hard part now is running the loop: GPUs, scheduling, weight sync, job lifecycles.

RL is a loop.

Sample from the current policy, grade the samples, update the weights, hand them to the sampler. Repeat for a few hundred steps.

Sample generate rollouts sampler GPU, inference Grade score each rollout CPU: tools, sandboxes, judges Train forward_backward, optim_step trainer GPU Sync weights trainer to sampler every step, to stay on-policy repeat Underneath every phase what the loop has to handle today inference servers GPU memory and placement loading each step's weights sandboxes and tool servers judge endpoints retries, timeouts, rate limits trainer processes, CUDA which GPU, whose turn checkpoints and resume storage and transport which weights the sampler holds recovery when a worker dies Each phase waits for the one before it, so a job on its own leaves its GPUs idle most of the time. Today all of this sits in the researcher's code. That coupling is the problem.

Decouple infra from AI research.

The researcher's code stays on top, OpenRL runs underneath, and the Tinker API is the contract between them.

AI researcher plain Python, on a laptop or in the cluster Dataset and environment rewards, tools, sandboxes RL algorithm GRPO, PPO, loss, advantages The loop itself tinker-cookbook, or hand-written Same loop. Any model, any accelerator. Tinker API: forward_backward, optim_step, save_weights, sample OpenRL self-hosted on your Kubernetes cluster API server, queue multi-tenant Trainers LoRA and full fine-tuning Samplers vLLM, delta weight sync Scheduler DRA, time-slicing GPUs shared across jobs; researchers never see them

The loop is user code.

Plain Python with the Tinker SDK. The environment, the reward, the loss function is domain specific.

import tinker
svc = tinker.ServiceClient(base_url="http://openrl.my-cluster:8000")
trainer = await svc.create_lora_training_client_async(
    base_model="Qwen/Qwen3-4B-Instruct-2507", rank=16)

for step in range(40):
    sampler = trainer.save_weights_and_get_sampling_client(name=f"step-{step}")
    rollouts = sampler.sample(prompt, num_samples=8, sampling_params=params).result()
    datums = score(rollouts)                      # your env, your reward
    await trainer.forward_backward_async(datums, loss_fn="importance_sampling")
    await trainer.optim_step_async(tinker.types.AdamParams(learning_rate=1e-5))
save_weights
Publish the trainer’s weights so the sampler sees them.
sample
Rollouts from the new weights, in parallel per prompt.
forward_backward
First half of a training step. You choose the loss.
optim_step
Second half. You choose the optimizer.

OpenRL provides:

Multi-tenant
Several RL post-training runs share one accelerator pool at the same time.
LoRA and full fine-tuning
Both behind the same API. LoRA runs on one base model share a trainer and a sampler as adapters, each with its own rank.
Shared accelerators
Full fine-tuning shares accelerators through llm-d's RL time-slicing primitives: state moves between HBM and DRAM so trainers and samplers take turns. Weights sync as sparse deltas, a few percent of the model per step.
Scheduler
Uses Kubernetes Dynamic Resource Allocation (DRA) to carve accelerator capacity into shapes, sizes each worker to one, and co-locates workers that can time-slice the same accelerator.
Tinker-compatible API
Use the Tinker SDK and the tinker-cookbook as they are; point ServiceClient at your cluster.

How OpenRL is built.

A control plane that takes requests and places workers. A data plane where trainers and samplers run.

Kubernetes cluster OpenRL control plane OpenRL data plane RL loop Tinker SDK many of them, on laptops or inside the cluster API server Tinker-compatible API over HTTP forward_backward, optim_step, save_weights, sample, plus checkpoints worker CRs Scheduler Go controller for worker life-cycle placement: spread out, or binpack onto shared accelerators creates and places workers enqueues each request Queue multi-tenant, Redis pull workers, one set per tenant Trainer workers forward_backward, optim_step Sampler workers sample writes weights each step reads them Shared filesystem checkpoints, and the weights that move from trainer to sampler every step

How a GPU is shared.

Full fine-tuning needs the whole device, so co-located workers take turns. A timeslicer grants the turn; llm-d's snapshot agent parks the state in between.

Kubernetes node joins the trainer or sampler pool with one label; workers land on it as pods Trainer, tenant A pod: forward_backward, optim_step Sampler, tenant A pod: sample Trainer, tenant B pod: forward_backward, optim_step acquire, yield Timeslicer node-local DaemonSet, one holder at a time workers reach it over a local socket 1 a worker asks to acquire 2 the holder yields; its state goes to DRAM 3 the next state returns to HBM; it runs llm-d snapshot agent parks accelerator state in DRAM and restores it for the next holder the holder computes Accelerator who holds it, over time A trains B trains A samples A trains B trains A samples snapshot restore Host DRAM parked state of the others A parked worker resumes from the same state when its turn comes back.

Concurrent RL in action.

Reproducing LoRA Without Regret on GSM8K RL: full fine-tuning against LoRA rank 32 and rank 1, two seeds each, six runs training at once on one OpenRL cluster.

Mean reward, last 10 of 40 steps

full fine-tuning
0.934
LoRA rank 32
0.957
LoRA rank 1
0.936

Seed-to-seed spread is 0.013 to 0.016. Rank 1 lands inside it.

All six trained at the same time. The four LoRA runs shared one trainer and one sampler as adapters, and the two full fine-tuning seeds were packed two to a claim. No run had a GPU to itself.

Qwen3-8B on H100, 40 steps of 64 episodes. Line: mean of two seeds, 5-step rolling mean. Band: the spread between the seeds. Data: docs/experiments/lora-without-regret on branch experiment/lora-without-regret. Paper: thinkingmachines.ai/blog/lora

Time-slicing lifts the duty cycle.

Three jobs on two shared H100s instead of six dedicated GPUs. Same learning curves, more than twice the trainer duty cycle.

Duty cycle over time for six dedicated baseline GPUs, each mostly idle, against two time-sliced GPUs that stay busy across all three jobs

Time-sliced, three jobs on two H100s

Trainer duty cycle
34.2%
GPUs provisioned
2
GPU-hours
1.30
Wall clock, all three
39 min

On six dedicated GPUs: 15.6% duty cycle and 2.10 GPU-hours. The same two GPUs running the jobs one after another: 54 minutes.

Context switches cost 0.5 to 1.6 s at the median; queue waits per phase run 1 to 15 s. All three jobs matched their dedicated-GPU learning curves.

Jobs: Text-to-SQL RL on Qwen3-1.7B, math RL on Qwen2.5-7B, dialogue SFT on Gemma 4 E2B. Figure from llm-d.ai/blog/increase-researcher-velocity-rl-llm-d-time-slicing

Few weights change per step.

Qwen3-8B on GSM8K math RL. The share of parameters that change in a step falls from 12% at step 1 to under 3% by step 25, while reward climbs. So the trainer ships only the changed parameters, and the sampler patches them in place.

Model weights mutation versus reward progression for Qwen3-8B on GSM8K math RL over 49 steps: the mutated share falls from 12.9% at step 1 to about 2.6% by step 25 and stays near 3%; reward rises from 0.33 to about 1.0 by step 10

Delta weight sync, Qwen3-8B

Weights changed, step 1
12%
Weights changed, step 50
2.6%
Bytes per sync
1.2 GB
Sampler weight load
5 s

A full checkpoint is 16 GB and took 35 s to load on the sampler. Deltas travel over shared NFS on GKE, so trainer and sampler need no RDMA link and can be placed independently.

The trainer already holds the previous weights in host memory for time-slicing; that is what the delta is computed against.

Figure from the note “Delta Weights in RL” (Jul 2026). Per-step density for the same model is in docs/experiments/lora-without-regret/delta_density.csv on branch experiment/lora-without-regret.

Getting started.

Three commands, and you have RL as a service on your own cluster. Then point the Tinker SDK at it.

Install

1

Create a GKE cluster on a recent release for DRA, with Filestore as the shared filesystem.

$ gcloud container clusters create ml \
    --location us-central1 --release-channel rapid \
    --addons GcpFilestoreCsiDriver
2

Add accelerator pools. The labels opt a node in and say which roles it takes; repeat with openrl.io/sampler=true.

$ gcloud container node-pools create trainers --cluster ml \
    --accelerator type=nvidia-l4,count=2 \
    --node-labels openrl.io/enabled=true,openrl.io/trainer=true
3

Install OpenRL.

$ kubectl apply -f https://github.com/gke-labs/open-rl/releases/latest/download/openrl-distributed-shared.yaml

Use

Point the Tinker SDK at it. The loop from Part 2 runs unchanged.

import tinker
# your cluster, not Tinker's cloud
svc = tinker.ServiceClient(
    base_url="http://openrl.my-cluster:8000")
trainer = await svc.create_lora_training_client_async(
    base_model="Qwen/Qwen3-4B-Instruct-2507", rank=16)
# then sample, score, forward_backward, optim_step

Or run a recipe, or hand your coding agent the tinker-cookbook and ask for a loop.

We have just started.

What runs today, and what is next in the six months ahead.

Production readiness

nowRuns our own fine-tuning workloads. Installs from one manifest.

nextDay-0 and day-2 operating guides. A versioned API with an upgrade path.

Multi-GPU per host

nowOne GPU per trainer or sampler worker.

nextSamplers first, since vLLM already does tensor parallelism. Then trainers. Then the Megatron engine.

More accelerators

nowNVIDIA L4 and H100 through Kubernetes DRA.

nextA TPU proof of concept, then an MVP.

More models

nowDense Qwen and Gemma to about 10B, LoRA and full fine-tuning.

nextDense models to 30B. MoE for training and sampling.

Observability

nowkubectl over Workloads, claims and pods.

nextA CLI agents can consume, a dashboard, per-step metrics in a standard format, cancelling a job.

Recipes

nowText-to-SQL RL, SFT notebooks, autoresearch, the tinker-cookbook as it is.

nextA Legal Agent Benchmark recipe, a finance domain, paper reproductions, the checkpoints API.

The full plan, by focus area and initiative, is in ROADMAP.md. To change it, open an issue with the roadmap label.

Build it with us.

We have applied to donate OpenRL to the CNCF, for a vendor-neutral home.

github.com/gke-labs/open-rl

The roadmap and the issues are public. File one, or pick one up.

Try it

  • Pig Latin SFT notebook on a laptop, no GPU required.
  • Text-to-SQL RL recipe for the full sample, grade, train loop.
  • Autoresearch recipes: parallel experiments against one API server.
  • GKE setup guide to stand up the shared cluster.

examples/sft/pig-latin · examples/text-to-sql · examples/autoresearch · docs/setup/gke-setup.md

Thank you.

Same loop. Any model, any accelerator.

Sunil Arora
droot@

github.com/gke-labs/open-rl

Pre-training gives you the base model.

Every capability we care about is already in it, at low probability.

Post-training (RL) sharpens what is already there.

RL raises the probability of what you reward.