CRISP reconstructs simulation-ready human motion and scene geometry from monocular RGB video. It fits planar scene primitives and uses human–scene contacts to recover occluded geometry, enabling physics-based humanoid motion tracking with reinforcement learning. In the paper's EMDB/PROX evaluations, CRISP reduces motion-tracking failure rates from 55.2% to 6.9% and improves RL simulation throughput by 43%.
Click and Drag / Zoom-in(out) interactive demo to see simulation result everywhere (when not shown please refresh page).
Page 1 / 1
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Input Video
Interactive (ours)
Inspect VideoMimic reconstructions next to CRISP and the original footage. Each card shows the input video alongside both interactive viewers for direct comparison.
Page 1 / 1
Input Video
CRISP (ours)
VideoMimic
Input Video
CRISP (ours)
VideoMimic
Input Video
CRISP (ours)
VideoMimic
Input Video
CRISP (ours)
VideoMimic
Input Video
CRISP (ours)
VideoMimic
Input Video
CRISP (ours)
VideoMimic
Input Video
CRISP (ours)
VideoMimic
Input Video
CRISP (ours)
VideoMimic
Input Video
CRISP (ours)
VideoMimic
Input Video
CRISP (ours)
VideoMimic
Input Video
CRISP (ours)
VideoMimic
Input Video
CRISP (ours)
VideoMimic
Input Video
CRISP (ours)
VideoMimic
Input Video
CRISP (ours)
VideoMimic
We introduce CRISP, a method that recovers simulatable human motion and scene geometry from monocular video. Prior work on joint human-scene reconstruction relies on data-driven priors and joint optimization with no physics in the loop, or recovers noisy geometry with artifacts that cause motion tracking policies with scene interactions to fail. In contrast, our key insight is to recover convex, clean, and simulation-ready geometry by fitting planar primitives to a point cloud reconstruction of the scene, via a simple clustering pipeline over depth, normals, and flow. To reconstruct scene geometry that might be occluded during interactions, we make use of human-scene contact modeling (e.g., we use human posture to reconstruct the occluded seat of a chair). Finally, we ensure that human and scene reconstructions are physically-plausible by using them to drive a humanoid controller via reinforcement learning. Our approach reduces motion tracking failure rates from 55.2% to 6.9% on human-centric video benchmarks (EMDB, PROX), while delivering a 43% faster RL simulation throughput. We further validate it on in-the-wild videos including casually-captured videos, Internet videos, and even Sora-generated videos. This demonstrates CRISP’s ability to generate physically-valid human motion and interaction environments at scale, greatly advancing real-to-sim applications for robotics and AR/VR. Code and interactive demos are available at our project website: crisp-real2sim.github.io/CRISP-Real2Sim.
On EMDB/PROX, CRISP’s contact-guided planar reconstruction reduces motion-tracking failures from 55.2% to 6.9% compared with VideoMimic under the same RL tracking protocol (Table 1).
Click and Drag / Zoom-in(out) inside each Viser viewer to orbit the scene and scroll to zoom. Press play to watch the simulation from your chosen angle.
View the scene-aware policy gallery/ Download
@inproceedings{wang2026contact,
title={Contact-guided real2sim from monocular video with planar scene primitives},
author={Wang, Zihan and Wang, Jiashun and Tan, Jeff and Zhao, Yiwen and Hodgins, Jessica and Tulsiani, Shubham and Ramanan, Deva},
booktitle={International Conference on Learning Representations},
volume={2026},
pages={115452--115468},
year={2026}
}We extend our deepest gratitude to Guanya Shi for invaluable feedback and insightful discussions on CRISP. We thank Qitao Zhao for writing suggestions, Arthur Allshire for helping set up baselines, Zhengyi Luo and Yuxuan Kuang for early stage discussions, Luna Shi and Weiyu Li for helpful discussions. The research project is funded by Bosch Research Center for Artificial Intelligence.