Dreaming the Sound of Contact

Recent advances in video generation enable learning robot manipulation trajectories from generated videos. However, these approaches produce purely kinematic trajectories that lack force information, leading to failure in contact-rich tasks where appropriate contact forces are essential for success. Generated audio carries a complementary and underexplored signal: contact sounds encode force dynamics that video alone cannot capture.

We present a pipeline that jointly leverages generated video and audio to recover both motion trajectories and contact force profiles from a single task description. We execute these force-aware trajectories on a Franka Panda robot using a closed-loop force regulator that tracks the audio-derived force profile during contact. Real-robot experiments across four contact-rich tasks — whiteboard wiping, carrot peeling, box stacking, and button pressing — demonstrate that our force-aware pipeline enables successful manipulation from video generation, where a kinematic-only baseline fails.

We further show that the pipeline can serve as an automated force-aware data-generation engine: a closed-loop policy trained on the generated demonstrations generalizes across object placements and appearances.

Beyond single-trajectory execution, we use the pipeline to generate training data: for all four tasks, we randomize the object position, run the full force-aware pipeline, and keep 50 successful rollouts per task as demonstrations, each recording end-effector pose, force profile, and two RGB views. We train a CNN-based Diffusion Policy from the wrist and external RGB images and the end-effector pose, in two variants: one without force input and one that additionally observes the measured 3-D contact force. At test time both command poses through the same fixed-stiffness impedance controller, with no force regulator, on unseen object placements and appearances.

The policy without force input already succeeds in 29/40 trials across the four tasks; adding force input raises this to 34/40, with the largest gains on peeling and pressing. Because the demonstrations are collected under force-aware execution, the recorded poses already encode the contact behavior: comparing all demonstrations with all successful rollouts as mean ± std over in-contact task progress (normalized end-effector path length accumulated during contact) shows both policies maintaining sustained contact and tracking the demonstrations within roughly 2 N on wiping, pressing, and stacking. Peeling is the exception: both policies press harder than the demonstrations, and the force-input policy presses hardest, which removes the skin more reliably.