WorldBench
Regression testing for robot world-model checkpoints
Did your new world-model checkpoint actually improve?
WorldBench evaluates baseline and candidate predictions on the same robot episodes, finds regressions across metrics, episodes, and prediction horizons, and returns a CI-friendly PASS or FAIL.
eval-batch
per-horizon curves
PASS / FAIL gates
Real checkpoint proof
NanoWM 50k baseline to 300k candidate on the same 10 RT-1 episodes.
WorldBench gate
NanoWM-B/2
Composite Score
85.67 → 87.28
Change
+1.61
Visual
+2.19
Temporal
+0.89
Episodes
9 improved, 1 regressed
Gate
PASS
The later checkpoint improved overall, while WorldBench still caught the episode where it got worse. This is a fixed 10-episode proof, not a leaderboard result.
Scored metrics: Visual Similarity and Temporal Stability only. Action Consistency, Object Permanence, and Contact Realism were N/A for these RGB video pairs.
1pip install "worldbench[video]"
2worldbench eval-batch --ground-truth eval_suite/ --predictions checkpoint_old/ --name checkpoint_old --config worldbench.yml
3worldbench eval-batch --ground-truth eval_suite/ --predictions checkpoint_new/ --name checkpoint_new --config worldbench.yml
4worldbench gate --baseline checkpoint_old.json --candidate checkpoint_new.json --config worldbench.yml01 - What WorldBench evaluates
Scores stay explicit about which evidence was actually available.
Your model code generates prediction videos. WorldBench evaluates them locally, keeps unsupported metrics as N/A, aggregates episode and horizon behavior, and returns PASS or FAIL.
Visual Similarity
Compares aligned ground-truth and predicted RGB future frames.
Temporal Stability
Measures future-frame transitions when at least two predicted frames exist.
Action Consistency
Requires known action semantics or a robot-specific action adapter.
Object Permanence
Requires reliable object tracking; otherwise it remains N/A.
Contact Realism
Requires robot and object tracking; otherwise it remains N/A.
Composite Score
Reports metric count, configured-weight coverage, effective weights, and unsupported metrics.
02 - CI regression gate
Fail a release when the candidate loses required evidence or quality.
WorldBench writes timestamped result JSON for video, batch, and gate runs; unsupported metrics stay N/A instead of being averaged as zero.
WorldBench Gate
gate.json
Status
PASS
Composite change
+1.61
Episodes
9 improved / 1 regressed
The candidate improved overall, but WorldBench still surfaced the one episode-level regression.
Read the docs
03 - Honest limitations
WorldBench is a local regression tool, not a universal robotics benchmark.
Metric coverage
The NanoWM proof supports two metrics. Raw actions and real scenes need adapters for the remaining metrics.
Validation scope
The public proof covers two checkpoints and 10 fixed episodes, not a universal ranking.
Saved predictions
WorldBench evaluates aligned prediction artifacts; teams provide their own model inference.
Looking for 3 early testing partners.
Training a robot world model? Send prediction videos from the same episodes for your baseline and candidate checkpoints. I will personally help run the first evaluation.