Teaching Whole-Body Dexterity with a Wearable Contact Interface
We evaluate two additional baselines on Carry-and-Push with 10 rollouts each, using the same demonstrations and evaluation configurations. The contact-gated fixed-link baseline uses the same contact positions and activation flags but assigns each goal to a fixed body part. The continuous tracking baseline tracks the same contact points at all times, without contact states, and retains the any-surface objective.
| Method | Contact Gating | Assignment | Success |
|---|---|---|---|
| Humanoid UMI++ (original) | No | Fixed link | 4/10 (40%) |
| Continuous tracking (new) | No | Any surface | 4/10 (40%) |
| Contact-gated fixed-link (new) | Yes | Fixed link | 5/10 (50%) |
| Ours (original) | Yes | Any surface | 9/10 (90%) |
The robot sometimes drops the carried box or fails to reach the floor box with its leg. Tracking wrist poses and fixed arm keypoints at the same time can prevent the arms from maintaining sufficient support. Fixed body assignments also limit how the robot reaches each contact goal. Differences in body proportions and tracking error can leave the robot surface short of the intended contact location.
Carried box slips
Leg does not establish the intended contact
The robot sometimes drops the carried box or walks in the wrong direction. Without contact states, the controller tracks the same points even during non-contact motion. These extra constraints can interfere with hand tracking and whole-body coordination, making it harder for the robot to support the box and move in the intended direction.
Carried box slips
Robot drifts away from the intended direction
These are stricter counts from the original 16 Sweep-and-Stow rollouts per method. Full end-to-end completion requires sweeping every object into the bin and fully closing the cabinet.
| Method | Full Sweep | Full Close | Full End-to-End |
|---|---|---|---|
| Humanoid UMI++ | 1/16 | 0/16 | 0/16 |
| Ours | 16/16 | 13/16 | 13/16 (81.3%) |
Ours fully completes 8/8 seen-object and 5/8 unseen-object rollouts.
We compare additional Carry-and-Push runs with offsets of δ = 0, 0.025, and 0.075 m against the original 0.05 m setting. At zero offset, the controller aims for the intended contact location without an additional pressing displacement. Smaller offsets can leave the robot unable to push the floor box sufficiently. Increasing the offset to 0.075 m places the virtual target farther into the object and produces a more aggressive forward push.
Zero offset: δ = 0 m
Smaller offset: δ = 0.025 m
Original setting: δ = 0.05 m
Larger offset: δ = 0.075 m
We also test the robot's hold while the Carry-and-Push controller runs. A person pushes directly on the box between the robot's arms. The robot maintains arm support and keeps holding the box throughout the shown perturbation.
Manual push while holding the box
Humanoid robots have many degrees of freedom and contact surfaces, yet most loco-manipulation research restricts manipulation to the hands and avoids other body contacts with the environment. To extend manipulation beyond the hands, we present PanTact, a system that learns humanoid whole-body dexterity, the capability to manipulate with any suitable body surface, directly from robot-free human demonstrations. Our key idea is to use contact as manipulation-centric information that transfers across embodiments. Contact locations and timing specify the interactions needed to accomplish a task, while the robot chooses how to coordinate its body to realize them within its own physical constraints. Specifically, PanTact captures, learns, and executes whole-body contact via: 1) a wearable, robot-free data collection interface that augments egocentric vision and bimanual UMI grippers with whole-body modular contact patches to record body contact locations and states; 2) a visuomotor policy that mitigates the egocentric observation gap and predicts whole-body contact goals; and 3) contact-aware whole-body control formulations that realize active contact goals using suitable robot surfaces. We deploy PanTact on a Unitree G1 humanoid without additional body contact sensors, demonstrating whole-body dexterity involving body support, pushing, sweeping, and load-sharing.
Egocentric-augmented bimanual UMI grippers plus modular contact patches on the forearms and knees record where, when, and how hard the demonstrator's body contacts the environment.
Robot-free demonstration. Head and wrist camera views recorded alongside live readings from the forearm and knee contact patches, captured in sync while the demonstrator performs the task.
A contact-predictive policy maps embodiment-agnostic observations to EEF trajectories and whole-body contact goals.
The controller realizes active contact goals with any suitable robot surface, rather than assigning each contact to a fixed link.
Differential IK QP jointly optimizing EEF and contact tasks. Requires no training data or simulation; limited to upper-body tasks with fixed legs.
Contact-aware whole-body IK whose dense reference is executed by the existing SONIC motion tracker. Enables lower-body contacts and locomotion.
RL goal-conditioned controller that directly tracks sparse EEF and contact goals, trained in simulation with hindsight contact labeling.
The robot sweeps tall objects from a counter into an open bin in a single stroke, then uses a lower-body surface to push the cabinet closed. Ours (WBIK + SONIC) reaches a 90.63% graded end-to-end score; Humanoid UMI++ 0%.
Ours · WBIK + SONICThird-person view (left) and synchronized robot egocentric view (right)
Seen object, different layout
Unseen object
Two seen objects
Two unseen objects
One object
Two objects
The robot grasps a small object with the gripper, then embraces a bulky object with its arms and moves both simultaneously. Ours (Upper-Body IK) reaches 85% success; Humanoid UMI 0%.
Ours · Upper-Body IKRollout 1
Rollout 2
The robot carries a box in both arms, walks towards a larger box on the floor, and pushes it with its shins and knees. Ours (WBIK + SONIC) reaches 90% success; Humanoid UMI++ 40%.
Ours · WBIK + SONICRollout 1
Rollout 2 · box on right