Open evaluation data · 23 October 2026
See all answers.
You can compare recorded predictions against human labels, investigate disagreements, or browse datasets in the LeRobot catalogue.
Comparison of the models
Out of 927 labelled episodes, 862 have a definitive label. The remaining 65 ambiguous community clips are presented here for your examination but are not included in the metrics scoring.
Inputs and comparison methods
The released RoboJev reads 10 evenly-spaced front-camera frames, as well as end-frames from other cameras if available. Frontier LLMs answered in a zero-shot manner using just four front-camera frames. clip-v1 means 4 frames, dense-v1 means 10 frames + other views, video means whole-video input.
SO-101 has 268 held-out runs, and RoboJev was trained on other ArmnetBench sessions. The unseen labs contain 459 runs from eight labs that we didn't use for training. The community contains 200 clips that are only labelled by one person, and 135 of them have a clear outcome. We show missing results with a dash and all previous model variants can be selected.
Metrics, paired bootstrap intervals and protocol details (JSON)
A sample of robots being evaluated on the Hub.
68,555 episodes sampled, 501 hours total. These are not a full census of Hugging Face; they are just catalogue entries. The predictions from the previous four-frame model are not included, and we have not regraded this catalogue.
Download catalogue JSONLLoading recorded results…