Robot Hand

Results at a Glance

  • As a first task, we trained an ACT policy to control a robot hand to firmly grasp a mug by the handle, lift and rotate the mug, and finally place the mug and release the grip. This task is difficult for a 1-DOF gripper, especially if the mug is heavy.
  • A subtle motion of the thumb successfully helps release the grip.
  • The hand successfully lifts the mug even when it contains a heavy unbalanced load.

A First Experiment

Fig 1. Fully autonomous ACT policy.
Note: subtle thumb motion to release the grip!
Fig 2. Same ACT policy .
The unbalanced load plus the mug weighs a total of 1.5 lbs.

We attached a 6-DOF Alt-Bionics Surge robot hand to a Trossen WidowX AI arm in place of the gripper. To teleoperate the hand, we used a Manus MetaGloves Pro glove. We also used the glove's absolute rotation angle to control the wrist joints -- joints 3,4,5 -- of the arm. For the remaining three arm joints, we used a Trossen leader arm. We trained an ACT model on our pick up mug dataset for 100K training steps. As shown in Fig 1, the model successfully learned to pick up and release the mug.

  • Thumb motion: As shown in Fig 1, the thumb performs a subtle motion to help release the mug.
  • Heavy objects: One advantage of a hand over a gripper is the ability to use leverage and to spread pressure, which helps the hand lift the mug even when it contains a heavy unbalanced load -- 1.5 lbs load plus mug -- as shown in Fig 2.
  • Teleoperation of the hand: To control the hand with the Manus glove required acquiring a dataset of glove-hand finger position correspondences and then performing a regression.
  • New robot modules: To control both the Surge hand and the WidowX arm simultaneously in lerobot_teleoperate.py, lerobot_record.py, and lerobot_rollout.py, new robot modules which have 6 arm plus 6 hand actions/states were built in our fork of the Trossen lerobot plugin repo. For teleoperation, the new leader module reads both the leader arm position and the Manus glove outputs.
Back to top

Implementation Details

Under construction!

Back to top