OrbiSim

Project Page

OrbiSim: World Models as Differentiable Physics Engines for Embodied Intelligence

OrbiSim redefines world models as a differentiable physics engine that unifies structured scene assets, neural dynamics, visual prediction, and downstream control.

Jiajian Li* Jingyuan Huang* Junru Gong* Qi Wang Xiaokang Yang Yunbo Wang†

MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science
Shanghai Jiao Tong University

* Equal contribution.

† Corresponding author.

arXiv

Code coming soon.

Push
Cable
Stack
Articulated
Drape

Abstract

World models, reframed as differentiable simulators

We present OrbiSim, a novel robotic simulation paradigm that redefines world models as a fully differentiable physics engine for embodied intelligence. Unlike prior world models that focus on unconstrained imagination in latent or visual domains, OrbiSim establishes a unified, physically-grounded pathway that bridges structured scene assets, neural dynamics, and downstream reinforcement learning.

By enabling end-to-end differentiability throughout the entire simulation loop—spanning from explicit state transitions to visual observation generation—OrbiSim supports tasks traditionally intractable for classical simulators, such as differentiable contact modeling, gradient-based policy optimization under sparse rewards, and intuitive physical inference.

Empirical results demonstrate that OrbiSim significantly outperforms state-of-the-art world models in both predictive fidelity and control performance. Furthermore, its consistent responsiveness to asset configurations and physical parameters suggests its potential as a differentiable tool for enhancing robot simulation and policy training.

Overview

The core pipeline of OrbiSim

The model couples asset-conditioned dynamics with state-guided vision, enabling analytical gradients through the simulation loop for system identification and policy optimization.

Overview figure of OrbiSim showing the asset-conditioned representation, decoupled dynamics and vision modules, and end-to-end differentiability for optimization.

Core Strengths

Three ideas shape the OrbiSim design

The architecture is built around a small set of principles that connect representation, prediction, and optimization in a single differentiable simulation pipeline.

General-purpose world representation

OrbiSim adopts an asset-conditioned representation interface that supports heterogeneous object types through appropriate state and geometry encodings, rather than being limited to a task-specific design.

Decoupled dynamics and vision

By decoupling the neural architecture into interlinked dynamics and rendering modules, OrbiSim simultaneously predicts precise physical states and high-fidelity visual observations and enables seamless integration with existing simulation platforms.

End-to-end differentiability

OrbiSim supports differentiable system identification from state or visual rollouts, while propagating task gradients through the simulation for downstream action and policy optimization.

Experiments

Generative fidelity and downstream control

We evaluate OrbiSim as both a generative world model and a differentiable execution engine, focusing on generative fidelity and physical consistency under varying configurations, as well as the benefits of differentiable gradient pathways for downstream reinforcement learning.

Generative and Physical Fidelity

Performance on benchmark manipulation tasks

As shown in the table, OrbiSim (Final) consistently achieves state-of-the-art performance across all metrics and horizons. Compared with AdaWorld and Vid2World, OrbiSim maintains superior temporal coherence and lower trajectory error, demonstrating a more robust alignment between physical dynamics and visual synthesis.

Method PSNR10 ↑ PSNR100 ↑ LPIPS10 ↓ LPIPS100 ↓ FVD ↓ TrajErr ↓
Vid2World 22.2014 17.8856 0.1312 0.2551 1750.1 0.6754
AdaWorld 26.6647 12.8346 0.1183 0.3482 1305.8 1.8597
Orbisim w/o Decoupling 27.9346 19.9510 0.1188 0.1799 689.1 0.8134
Orbisim w/o Random Sampling 26.6890 19.1119 0.1076 0.1669 531.2 0.5742
Orbisim w/o Object-Centric 25.9373 19.7581 0.1123 0.1463 524.5 0.4687
Orbisim (Final) 26.7105 19.9819 0.1078 0.1428 533.9 0.4468

We report PSNR and LPIPS at different rollout horizons (10 / 100 steps), together with the overall FVD score. TrajErr measures the discrepancy between inferred physical states from generated videos and the corresponding true trajectories. All models perform autoregressive rollouts from shared initial states.

Rollouts across pushing, cable lifting, stacking, articulation, and draping

We visualize five physics-rich settings used throughout the paper: robotsuite Push under varying friction, MuJoCo Cable Lift, Isaac Lab Stack, AdaManip Articulated, and Physion Drape. Together, these rollouts highlight sensitivity to physical parameters, differentiable control over deformable cables, long-horizon stability, joint-constrained part motion, and geometry-conditioned cloth deformation under the same asset-conditioned simulation framework.

Robotsuite Push fixes the same initial visual observation and replays the same action sequence under different friction settings. OrbiSim responds to the changed physical parameter with distinct, physically consistent rollouts.
GT / High friction
GT / Low friction
OrbiSim / High friction
OrbiSim / Low friction

Downstream Control

Differentiability evaluation on policy optimization

While the sparse episodic reward design makes credit assignment particularly challenging, OrbiSim differs from traditional black-box simulators by exposing analytical gradient pathways that propagate task-specific reward signals directly to the policy parameters. As shown in the training curves and rollout comparisons below, OrbiSim achieves superior performance and convergence speed compared with model-free, model-based, and imitation baselines.

Training curves on the robotsuite Push task

In the robotsuite Push task, the goal is to push the first cube into the second one so that, after the collision, the second cube comes to rest as close as possible to the left table edge without falling off.

The reward is decomposed into three terms: r1 encourages the end effector to approach the first cube, r2 encourages the first cube to move into the second cube, and r3 encourages the second cube to settle near the left edge while remaining on the table.

Training curves on the robotsuite Push task with a shared legend and four subplots for r1, r2, r3, and total reward.
The x-axis denotes training episodes and the y-axis denotes normalized episode rewards defined in the paper, namely r1, r2, r3, and the total reward.

Rollout comparison on the robotsuite Push task

OrbiSim
DreamerV3
Behavior Cloning
SAC
PPO
PPO+RND

Policies trained with model-free RL fail to learn effective behaviors for the downstream task, while DreamerV3 and behavior cloning remain less stable under long-horizon interactions. In contrast, OrbiSim produces coherent and goal-directed behaviors across different scenarios.

Real-to-Sim Physical Inference and Real-Robot Policy Optimization

Policy Learning via Differentiable Physical-Property Prediction

In Cable Lift, two robot arms move designated points on a deformable cable toward target locations. We alternate physical-parameter identification and control optimization: state- or image-trajectory discrepancies update the cable parameters, then a differentiable OrbiSim-Dynamics rollout optimizes the action sequence using terminal keypoint distance. On the real AgileX dual-arm platform, RGB observations and recorded robot motion support vision-based SysID before the optimized actions are executed on cables with different material properties.

Simulation system identification

Simulation SysID results on the Cable Lift task. Entries are mean absolute errors of the final estimated cable parameters within each evaluation split. The bend_passes column reports a discrete iteration-count error; the other columns use their respective simulator parameter units.
Setting bend bend_passes recovery damping
State, ID 0.0025 0.10 0.0061 0.0093
State, OOD 0.0055 0.55 0.0238 0.0116
Vision, ID 0.0088 0.90 0.0185 0.0366
Vision, OOD 0.0196 1.80 0.0778 0.0487

Terminal keypoint-to-target distance

Final keypoint-to-target distance on the Cable Lift task. Simulation entries report the mean sum of the two keypoint-to-target distances after MuJoCo replay in meters, separately for ID and OOD settings. The real-world entry reports the mean distance in pixels. A dash indicates that the corresponding method was not evaluated in that setting.
Method Simulation (m) Real world (px)
ID OOD
SAC 0.4402 0.3678 –
π0.5 0.1124 0.0881 62.2
OrbiSim (state-based SysID) 0.0122 0.0153 –
OrbiSim (vision-based SysID) 0.0240 0.0290 17.5

Real-robot execution on two cables

OrbiSim / Cable 1
π0.5 / Cable 1
OrbiSim / Cable 2
π0.5 / Cable 2

The OrbiSim-optimized actions achieve a mean final keypoint-to-target distance of 17.5 ± 6.1 pixels, compared with 62.2 ± 36.3 pixels for π0.5. The optimized actions transfer zero-shot to the real robot without direct cable-state measurement.

Citation

BibTeX

@misc{li2026orbisimworldmodelsdifferentiable,
      title={OrbiSim: World Models as Differentiable Physics Engines for Embodied Intelligence},
      author={Jiajian Li and Jingyuan Huang and Junru Gong and Qi Wang and Xiaokang Yang and Yunbo Wang},
      year={2026},
      eprint={2605.16395},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2605.16395},
}