WorldCanvas: A Promptable Framework for Rich, User-Directed Simulations

WorldCanvas is a framework that enables rich, user-directed simulations by combining text, trajectories, and reference images. It allows for the generation of coherent, controllable events with multi-agent interactions, object entry/exit, and reference-guided appearance.

Details

WorldCanvas is a framework that combines text, trajectories, and reference images to enable rich, user-directed simulations. Unlike text-only approaches and existing trajectory-controlled image-to-video methods, this multimodal approach encodes motion, timing, and visibility through trajectories, while using natural language for semantic intent and reference images for visual grounding of object identity. This allows for the generation of coherent, controllable events that include multi-agent interactions, object entry/exit, and reference-guided appearance, as well as counterintuitive events. The resulting videos demonstrate not only temporal coherence but also emergent consistency, preserving object identity and scene despite temporary disappearance. By supporting expressive world events generation, WorldCanvas advances world models from passive predictors to interactive, user-shaped simulators.